Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 5/11/2016 2:12 AM, Edward Tomasz Napierała wrote:

On 0510T1522, Matthew Grooms wrote:


The PR 204901 filed for this can be closed now that the author (ahem)
has committed support for the camcontrol reprobe command ...

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204901
https://docs.freebsd.org/cgi/getmsg.cgi?fetch=866020+0+current/svn-src-head


Ugh.  I honestly have no idea how did I manage to miss your patch.
I did remember the discussion, I remember asking mav@ about what's the
best way to hook it up to CAM, but until yesterday I just didn't know
the patch (and the PR) existed.  Sorry for that.  Guess that's what
happens when I try to keep up with too many unrelated subprojects
at the same time.



No worries. Thanks for getting support for this in the tree. I look 
forward to not rebooting production VMs after a VMDK resize.


And while on the subject of VMware+SCSI, it would be great if someone 
could get the VMware PVSCSI driver into the tree for the 11.0 release. 
It's sitting in phabricator in review and I'm sure it would be useful to 
a bunch of FreeBSD users ...


https://reviews.freebsd.org/D4112

Thanks,

-Matthew
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Resizing a zpool as a VMware ESXi guest ...

[Edward Tomasz Napierała]

On 0510T1522, Matthew Grooms wrote:
> On 11/28/2015 10:03 PM, Matthew Grooms wrote:
> > On 11/28/2015 6:10 PM, Matthew Grooms wrote:
> >> On 11/27/2015 7:44 PM, Matthew Grooms wrote:
> >>> I spent the day looking over the FreeBSD cam and scsi_da source code.
> >>> After sprinkling a bunch of printf's around to see what code paths
> >>> were being called, It's obvious that Edward was correct in assuming
> >>> that ESXi doesn't return any 'Unit Attention' sense information in
> >>> response to a 'Read Capacity' request. This kinda makes sense as ESXi
> >>> emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09
> >>> ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't
> >>> defined until the SCSCI-3 spec. It's frustrating that the only way to
> >>> get the scsci_da code to call reprobe() is by receiving a command
> >>> from the device. Would something like this work? ...
> >>>
> >>> 1) Register a callback using xpt_register_async( daasync,
> >>> AC_REPROBE_DEVICE, path ) that calls reprobe()
> >>> 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the
> >>> bus:target:lun as the argument
> >>> 3) have cam_xpt capture the IOCTL request and call xpt_async(
> >>> AC_REPROBE_DEVICE, path ) as a result
> >>>
> >>> This way users would have the option of manually asking cam to
> >>> communicate the new size to geom. The only option now is one or more
> >>> reboots to gain access to the increased disk capacity. If this sounds
> >>> like a reasonable approach, I'll take a stab at implementing it.
> >>>
> >>
> >> Here is a proof of concept patch. I'm a complete noob when it comes to
> >> cam, scsi or freebsd kernel development for that matter, so I'm sure
> >> it could have been done a better way. In any case, I added a new
> >> command to camcontrol that allows you to specify a bus, target and lun
> >> as an argument. For example ...
> >>
> >> # camcontrol readcap da1 -h
> >> Device Size: 32 G, Block Length: 512 bytes
> >>
> >> # gpart show da1
> >> =>  40  58720176  da1  GPT  (28G)
> >> 40  587201761  freebsd-ufs  (28G)
> >>
> >> Note, I resized the VMDK disk in ESXi. The camcontrol output shows the
> >> size as 32G but geom thinks its 28G.
> >>
> >> # camcontrol devlist
> >>at scbus1 target 0 lun 0 (cd0,pass0)
> >>   at scbus2 target 0 lun 0 (pass1,da0)
> >>   at scbus2 target 1 lun 0 (pass2,da1)
> >>  at scbus3 target 0 lun 0 (da2,pass3)
> >>
> >> # camcontrol reprobe 2:1:0
> >>
> >> This generates an event that is captured by the scsci da device to
> >> forces a reprobe. The kernel output looks almost identical to when the
> >> 'Unit Attention' sense data is received ...
> >>
> >> Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
> >> Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
> >> Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes
> >> or `gpart undo da1` to revert them.
> >>
> >> Now that geom knows about the increased disk capacity, I can increase
> >> the partition size and grow the fs ...
> >>
> >> [root@iscsi-i /home/mgrooms]# gpart show da1
> >> =>  40  67108784  da1  GPT  (32G)
> >> 40  587201761  freebsd-ufs  (28G)
> >>   58720216   8388608   - free -  (4.0G)
> >>
> >> # gpart resize -i 1 da1
> >> da1p1 resized
> >>
> >> # growfs da1p1
> >> Device is mounted read-write; resizing will result in temporary write
> >> suspension for /var/data1.
> >> It's strongly recommended to make a backup before growing the file
> >> system.
> >> OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB
> >> to 32GB? [Yes/No] Yes
> >> super-block backups (for fsck_ffs -b #) at:
> >>  58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672
> >>
> >> # df -h
> >> FilesystemSizeUsed   Avail Capacity  Mounted on
> >> /dev/da0p3 18G5.3G 12G31%/
> >> devfs 1.0K1.0K  0B   100%/dev
> >> /dev/da1p1 31G 32M 28G 0%/var/data1
> >> /dev/da2p1 15G 32M 14G 0%/var/data2
> >>
> >> Sure would be nice to have something like this in the tree. It's
> >> really a drag to have to reboot production VMs to increase disk
> >> capacity when it could be easily avoided. I'm not sure what the
> >> correct IOCTL should look like. Maybe CAMIOCOMMAND is a better way to
> >> go? If someone with some experience with the cam/scsi subsystems was
> >> willing to give me some direction I'd be willing to try and rewrite
> >> the patch in a way that would be commit worthy. I just need some
> >> direction.
> >>
> >
> > Ok, last post until I get some feedback. Here's a new version of the
> > patch complete with man page updates. It communicates via CAMIOCOMMAND
> > instead of introducing a new ioctl value. I tried to model it after the
> > device reset option, hopefully with some degree of success. Functionally
> > it should be the same as the first patch.
> >
> 
> The PR 

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 11/28/2015 10:03 PM, Matthew Grooms wrote:

On 11/28/2015 6:10 PM, Matthew Grooms wrote:

On 11/27/2015 7:44 PM, Matthew Grooms wrote:

I spent the day looking over the FreeBSD cam and scsi_da source code.
After sprinkling a bunch of printf's around to see what code paths
were being called, It's obvious that Edward was correct in assuming
that ESXi doesn't return any 'Unit Attention' sense information in
response to a 'Read Capacity' request. This kinda makes sense as ESXi
emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09
ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't
defined until the SCSCI-3 spec. It's frustrating that the only way to
get the scsci_da code to call reprobe() is by receiving a command
from the device. Would something like this work? ...

1) Register a callback using xpt_register_async( daasync,
AC_REPROBE_DEVICE, path ) that calls reprobe()
2) Implement a new IOCTL in cam_xpt that camcontrol can call with the
bus:target:lun as the argument
3) have cam_xpt capture the IOCTL request and call xpt_async(
AC_REPROBE_DEVICE, path ) as a result

This way users would have the option of manually asking cam to
communicate the new size to geom. The only option now is one or more
reboots to gain access to the increased disk capacity. If this sounds
like a reasonable approach, I'll take a stab at implementing it.



Here is a proof of concept patch. I'm a complete noob when it comes to
cam, scsi or freebsd kernel development for that matter, so I'm sure
it could have been done a better way. In any case, I added a new
command to camcontrol that allows you to specify a bus, target and lun
as an argument. For example ...

# camcontrol readcap da1 -h
Device Size: 32 G, Block Length: 512 bytes

# gpart show da1
=>  40  58720176  da1  GPT  (28G)
40  587201761  freebsd-ufs  (28G)

Note, I resized the VMDK disk in ESXi. The camcontrol output shows the
size as 32G but geom thinks its 28G.

# camcontrol devlist
   at scbus1 target 0 lun 0 (cd0,pass0)
  at scbus2 target 0 lun 0 (pass1,da0)
  at scbus2 target 1 lun 0 (pass2,da1)
 at scbus3 target 0 lun 0 (da2,pass3)

# camcontrol reprobe 2:1:0

This generates an event that is captured by the scsci da device to
forces a reprobe. The kernel output looks almost identical to when the
'Unit Attention' sense data is received ...

Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes
or `gpart undo da1` to revert them.

Now that geom knows about the increased disk capacity, I can increase
the partition size and grow the fs ...

[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  67108784  da1  GPT  (32G)
40  587201761  freebsd-ufs  (28G)
  58720216   8388608   - free -  (4.0G)

# gpart resize -i 1 da1
da1p1 resized

# growfs da1p1
Device is mounted read-write; resizing will result in temporary write
suspension for /var/data1.
It's strongly recommended to make a backup before growing the file
system.
OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB
to 32GB? [Yes/No] Yes
super-block backups (for fsck_ffs -b #) at:
 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672

# df -h
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/da0p3 18G5.3G 12G31%/
devfs 1.0K1.0K  0B   100%/dev
/dev/da1p1 31G 32M 28G 0%/var/data1
/dev/da2p1 15G 32M 14G 0%/var/data2

Sure would be nice to have something like this in the tree. It's
really a drag to have to reboot production VMs to increase disk
capacity when it could be easily avoided. I'm not sure what the
correct IOCTL should look like. Maybe CAMIOCOMMAND is a better way to
go? If someone with some experience with the cam/scsi subsystems was
willing to give me some direction I'd be willing to try and rewrite
the patch in a way that would be commit worthy. I just need some
direction.



Ok, last post until I get some feedback. Here's a new version of the
patch complete with man page updates. It communicates via CAMIOCOMMAND
instead of introducing a new ioctl value. I tried to model it after the
device reset option, hopefully with some degree of success. Functionally
it should be the same as the first patch.



The PR 204901 filed for this can be closed now that the author (ahem) 
has committed support for the camcontrol reprobe command ...


https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204901
https://docs.freebsd.org/cgi/getmsg.cgi?fetch=866020+0+current/svn-src-head

-Matthew
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 11/27/2015 7:44 PM, Matthew Grooms wrote:
I spent the day looking over the FreeBSD cam and scsi_da source code. 
After sprinkling a bunch of printf's around to see what code paths 
were being called, It's obvious that Edward was correct in assuming 
that ESXi doesn't return any 'Unit Attention' sense information in 
response to a 'Read Capacity' request. This kinda makes sense as ESXi 
emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09 
ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't 
defined until the SCSCI-3 spec. It's frustrating that the only way to 
get the scsci_da code to call reprobe() is by receiving a command from 
the device. Would something like this work? ...


1) Register a callback using xpt_register_async( daasync, 
AC_REPROBE_DEVICE, path ) that calls reprobe()
2) Implement a new IOCTL in cam_xpt that camcontrol can call with the 
bus:target:lun as the argument
3) have cam_xpt capture the IOCTL request and call xpt_async( 
AC_REPROBE_DEVICE, path ) as a result


This way users would have the option of manually asking cam to 
communicate the new size to geom. The only option now is one or more 
reboots to gain access to the increased disk capacity. If this sounds 
like a reasonable approach, I'll take a stab at implementing it.




Here is a proof of concept patch. I'm a complete noob when it comes to 
cam, scsi or freebsd kernel development for that matter, so I'm sure it 
could have been done a better way. In any case, I added a new command to 
camcontrol that allows you to specify a bus, target and lun as an 
argument. For example ...


# camcontrol readcap da1 -h
Device Size: 32 G, Block Length: 512 bytes

# gpart show da1
=>  40  58720176  da1  GPT  (28G)
40  587201761  freebsd-ufs  (28G)

Note, I resized the VMDK disk in ESXi. The camcontrol output shows the 
size as 32G but geom thinks its 28G.


# camcontrol devlist
   at scbus1 target 0 lun 0 (cd0,pass0)
  at scbus2 target 0 lun 0 (pass1,da0)
  at scbus2 target 1 lun 0 (pass2,da1)
 at scbus3 target 0 lun 0 (da2,pass3)

# camcontrol reprobe 2:1:0

This generates an event that is captured by the scsci da device to 
forces a reprobe. The kernel output looks almost identical to when the 
'Unit Attention' sense data is received ...


Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes 
or `gpart undo da1` to revert them.


Now that geom knows about the increased disk capacity, I can increase 
the partition size and grow the fs ...


[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  67108784  da1  GPT  (32G)
40  587201761  freebsd-ufs  (28G)
  58720216   8388608   - free -  (4.0G)

# gpart resize -i 1 da1
da1p1 resized

# growfs da1p1
Device is mounted read-write; resizing will result in temporary write 
suspension for /var/data1.

It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB to 
32GB? [Yes/No] Yes

super-block backups (for fsck_ffs -b #) at:
 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672

# df -h
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/da0p3 18G5.3G 12G31%/
devfs 1.0K1.0K  0B   100%/dev
/dev/da1p1 31G 32M 28G 0%/var/data1
/dev/da2p1 15G 32M 14G 0%/var/data2

Sure would be nice to have something like this in the tree. It's really 
a drag to have to reboot production VMs to increase disk capacity when 
it could be easily avoided. I'm not sure what the correct IOCTL should 
look like. Maybe CAMIOCOMMAND is a better way to go? If someone with 
some experience with the cam/scsi subsystems was willing to give me some 
direction I'd be willing to try and rewrite the patch in a way that 
would be commit worthy. I just need some direction.


Thanks,

-Matthew
Index: lib/libcam/camlib.c
===
--- lib/libcam/camlib.c (revision 291390)
+++ lib/libcam/camlib.c (working copy)
@@ -752,3 +752,41 @@
bcopy(src, dst, sizeof(struct cam_device));
 
 }
+
+/*
+ * Send a reprobe unit request for a given bus, target and lun
+ */
+int
+cam_reprobe_btl(path_id_t path_id, target_id_t target_id, lun_id_t target_lun)
+{
+   int fd;
+   char *func_name = "cam_reprobe_btl";
+   union ccb ccb;
+
+   if ((fd = open(XPT_DEVICE, O_RDWR)) < 0) {
+   snprintf(cam_errbuf, CAM_ERRBUF_SIZE,
+"%s: couldn't open %s\n%s: %s", func_name, XPT_DEVICE,
+func_name, strerror(errno));
+   return(-1);
+   }
+
+   /* Setup our request ccb */
+   bzero(_h, sizeof(struct ccb_hdr));
+   ccb.ccb_h.path_id = path_id;
+   

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 11/28/2015 6:10 PM, Matthew Grooms wrote:

On 11/27/2015 7:44 PM, Matthew Grooms wrote:
I spent the day looking over the FreeBSD cam and scsi_da source code. 
After sprinkling a bunch of printf's around to see what code paths 
were being called, It's obvious that Edward was correct in assuming 
that ESXi doesn't return any 'Unit Attention' sense information in 
response to a 'Read Capacity' request. This kinda makes sense as ESXi 
emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09 
ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't 
defined until the SCSCI-3 spec. It's frustrating that the only way to 
get the scsci_da code to call reprobe() is by receiving a command 
from the device. Would something like this work? ...


1) Register a callback using xpt_register_async( daasync, 
AC_REPROBE_DEVICE, path ) that calls reprobe()
2) Implement a new IOCTL in cam_xpt that camcontrol can call with the 
bus:target:lun as the argument
3) have cam_xpt capture the IOCTL request and call xpt_async( 
AC_REPROBE_DEVICE, path ) as a result


This way users would have the option of manually asking cam to 
communicate the new size to geom. The only option now is one or more 
reboots to gain access to the increased disk capacity. If this sounds 
like a reasonable approach, I'll take a stab at implementing it.




Here is a proof of concept patch. I'm a complete noob when it comes to 
cam, scsi or freebsd kernel development for that matter, so I'm sure 
it could have been done a better way. In any case, I added a new 
command to camcontrol that allows you to specify a bus, target and lun 
as an argument. For example ...


# camcontrol readcap da1 -h
Device Size: 32 G, Block Length: 512 bytes

# gpart show da1
=>  40  58720176  da1  GPT  (28G)
40  587201761  freebsd-ufs  (28G)

Note, I resized the VMDK disk in ESXi. The camcontrol output shows the 
size as 32G but geom thinks its 28G.


# camcontrol devlist
   at scbus1 target 0 lun 0 (cd0,pass0)
  at scbus2 target 0 lun 0 (pass1,da0)
  at scbus2 target 1 lun 0 (pass2,da1)
 at scbus3 target 0 lun 0 (da2,pass3)

# camcontrol reprobe 2:1:0

This generates an event that is captured by the scsci da device to 
forces a reprobe. The kernel output looks almost identical to when the 
'Unit Attention' sense data is received ...


Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes 
or `gpart undo da1` to revert them.


Now that geom knows about the increased disk capacity, I can increase 
the partition size and grow the fs ...


[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  67108784  da1  GPT  (32G)
40  587201761  freebsd-ufs  (28G)
  58720216   8388608   - free -  (4.0G)

# gpart resize -i 1 da1
da1p1 resized

# growfs da1p1
Device is mounted read-write; resizing will result in temporary write 
suspension for /var/data1.
It's strongly recommended to make a backup before growing the file 
system.
OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB 
to 32GB? [Yes/No] Yes

super-block backups (for fsck_ffs -b #) at:
 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672

# df -h
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/da0p3 18G5.3G 12G31%/
devfs 1.0K1.0K  0B   100%/dev
/dev/da1p1 31G 32M 28G 0%/var/data1
/dev/da2p1 15G 32M 14G 0%/var/data2

Sure would be nice to have something like this in the tree. It's 
really a drag to have to reboot production VMs to increase disk 
capacity when it could be easily avoided. I'm not sure what the 
correct IOCTL should look like. Maybe CAMIOCOMMAND is a better way to 
go? If someone with some experience with the cam/scsi subsystems was 
willing to give me some direction I'd be willing to try and rewrite 
the patch in a way that would be commit worthy. I just need some 
direction.




Ok, last post until I get some feedback. Here's a new version of the 
patch complete with man page updates. It communicates via CAMIOCOMMAND 
instead of introducing a new ioctl value. I tried to model it after the 
device reset option, hopefully with some degree of success. Functionally 
it should be the same as the first patch.


Thanks,

-Matthew
Index: sbin/camcontrol/camcontrol.8
===
--- sbin/camcontrol/camcontrol.8(revision 291390)
+++ sbin/camcontrol/camcontrol.8(working copy)
@@ -104,6 +104,9 @@
 .Ic reset
 .Aq all | bus Ns Op :target:lun
 .Nm
+.Ic reprobe
+.Aq bus:target:lun
+.Nm
 .Ic defects
 .Op device id
 .Op generic args
@@ -548,6 +551,9 @@
 connecting to that device.
 Note that this can have a destructive impact
 on the system.
+.It Ic reprobe
+Tell the kernel to re-probe the given 

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 11/27/2015 12:08 PM, Matthew Grooms wrote:

On 11/27/2015 3:16 AM, Willem Jan Withagen wrote:

On 27-11-2015 06:59, Matthew Grooms wrote:

All,

I know this is a very late follow up, but spent some more time looking
at this today and found some additional information that I found quite
interesting. I setup two VMs, one that acts as an iSCSI initiator (
CURRENT ) and another that acts as a target ( 10.2-RELEASE ). Both are
running under ESXi v5.5. There are two block devices on the initiator,
da1 and da2, that I used for resize testing ...

[root@iscsi-i /home/mgrooms]# camcontrol devlist
   at scbus1 target 0 lun 0 (cd0,pass0)
  at scbus2 target 0 lun 0 (pass1,da0)
  at scbus2 target 1 lun 0 (pass2,da1)
 at scbus3 target 0 lun 0 (da2,pass3)

The da1 device is a virtual disk hanging off of a VMware virtual SAS
controller ...

[root@iscsi-i /home/mgrooms]# pciconf
...
mpt0@pci0:3:0:0:class=0x010700 card=0x197615ad chip=0x00541000
rev=0x01 hdr=0x00
 vendor = 'LSI Logic / Symbios Logic'
 device = 'SAS1068 PCI-X Fusion-MPT SAS'
 class  = mass storage
 subclass   = SAS

[root@iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  20971440  da1  GPT  (10G)
 40  209714401  freebsd-ufs  (10G)

The da2 device is an iSCSI LUN mounted from my FreeBSD 10.2 VM running
ctld ...

[root@iscsi-i /home/mgrooms]# iscsictl
Target name  Target portalState
iqn.2015-01.lab.shrew:target0iscsi-t.shrew.lab Connected: da2

[root@iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da2
=>  40  20971440  da2  GPT  (10G)
 4024   - free -  (12K)
 64  209713921  freebsd-ufs  (10G)
   2097145624   - free -  (12K)

When I increased the size of da1 ( the VMDK ) and then re-ran
'camcontrol readcap' without a reboot, it clearly showed that the disk
size had increased. However, geom failed to recognize the additional
capacity ...

[root@iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
Device Size: 16 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  20971440  da1  GPT  (10G)
 40  209714401  freebsd-ufs  (10G)

Here is the interesting bit. I increased the size of da2 by modifying
the lun size in ctld.conf on the target and then issued a 
/etc/rd.d/ctld

reload. When I re-ran 'camcontrol readcap' on the initiator without a
reboot, it also showed that the disk size had increased, but this time
geom recognized the additional capacity as well ...

[root@iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
Device Size: 16 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da2
=>  40  33554352  da2  GPT  (16G)
 4024   - free -  (12K)
 64  209713921  freebsd-ufs  (10G)
   20971456  12582936   - free -  (6.0G)

I was then able to resize the partition and then grow the UFS
filesystem, all without rebooting the VM ...

[root@iscsi-i /home/mgrooms]# gpart resize -i 1 da2
da2p1 resized

[root@iscsi-i /home/mgrooms]# gpart show da2
=>  40  33554352  da2  GPT  (16G)
 4024   - free -  (12K)
 64  335543041  freebsd-ufs  (16G)
   3355436824   - free -  (12K)

[root@iscsi-i /home/mgrooms]# growfs da2p1
Device is mounted read-write; resizing will result in temporary write
suspension for /var/data2.
It's strongly recommended to make a backup before growing the file 
system.
OK to grow filesystem on /dev/da2p1, mounted on /var/data2, from 
10GB to

16GB? [Yes/No] Yes
super-block backups (for fsck_ffs -b #) at:
  21798272, 23080512, 24362752, 25644992, 26927232, 28209472, 29491712,
30773952, 32056192, 8432

[root@iscsi-i /home/mgrooms]# df -h
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/da0p3 15G1.2G 12G 9%/
devfs 1.0K1.0K  0B   100%/dev
/dev/da1p19.7G 32M8.9G 0%/var/data1
/dev/da2p1 15G 32M 14G 0%/var/data2

It's also worth noting that the additional space was not recognized by
gpart/geom on the initiator until after the 'camcontrol readcap da2'
command was run. In other words, I'm skeptical that it was a Unit
Attention notification that made the right thing happen since it still
took manual prodding of cam to get the new disk geometry up into the
geom layer.


I remember doing this for a bhyve VM, and had the type same problem.
Getting gpart in the VM to actually pickup the new size required some
extra prodding (I like that word) or rebooting the VM.
I can remember reporting this:
tpoic: "resampeling of a ZVOL that has been resized"
and getting a fix from Andrey V. Elsukov...

Index: head/sys/geom/part/g_part_gpt.c

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 11/27/2015 3:16 AM, Willem Jan Withagen wrote:

On 27-11-2015 06:59, Matthew Grooms wrote:

On 10/16/2014 3:10 AM, Edward Tomasz Napierała wrote:

On 1010T1529, Matthew Grooms wrote:

All,

I am a long time user and advocate of FreeBSD and manage a several
deployments of FreeBSD in a few data centers. Now that these
environments are almost always virtual, it would make sense that 
FreeBSD
support for basic features such as dynamic disk resizing. It looks 
like
most of the parts are intended to work. Kudos to the FreeBSD 
foundation

for seeing the need and sponsoring dynamic increase of online UFS
filesystems via growfs. Unfortunately, it would appear that there are
still problems in this area, such as ...

a) cam/geom recognizing when a drive's size has increased
b) zpool recognizing when a gpt partition size has increased

For example, if I do an install of FreeBSD 10 on VMware using ZFS, 
I see

the following ...

root@zpool-test:~ #  gpart show
=>  34  16777149  da0  GPT  (8.0G)
  34  10241  freebsd-boot  (512K)
1058   41943042  freebsd-swap  (2.0G)
 4195362  125818213  freebsd-zfs  (6.0G)

If I increase the VM disk size using VMware to 16G and rescan using
camcontrol, this is what I see ...

"camcontrol rescan" does not force fetching the updated disk size.
AFAIK there is no way to do that.  However, this should happen
automatically, if the "other side" properly sends proper Unit Attention
after resizing.  No idea why this doesn't happen with VMWare.
Reboot obviously clears things up.

[..]


Now I want the claim the additional 14 gigs of space for my zpool ...

root@zpool-test:~ # zpool status
pool: zroot
   state: ONLINE
scan: none requested
config:

  NAME STATE READ
WRITE CKSUM
  zroot ONLINE 0
0 0
gptid/352086bd-50b5-11e4-95b8-0050569b2a04 ONLINE 0
0 0

root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  5.97G   876M  5.11G14%  1.00x  ONLINE  -

The zpool appears to still only have 5.11G free. Lets reboot and try
again ...
Interesting.  This used to work; actually either of those 
(autoexpand or

online -e) should do the trick.


root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  14.0G   876M  13.1G 6%  1.00x  ONLINE  -

Now I have 13.1G free. I can add this space to any of my zfs 
volumes and

it picks the change up immediately. So the question remains, why do I
need to reboot the OS twice to allocate new disk space to a volume?
FreeBSD is first and foremost a server operating system. Servers are
commonly deployed in data centers. Virtual environments are now
commonplace in data centers, not the exception to the rule. VMware 
still

has the vast majority of the private virutal environment market. I
assume that most would expect things like this to work out of the box.
Did I miss a required step or is this fixed in CURRENT?

Looks like genuine bugs (or rather, one missing feature and one bug).
Filling PRs for those might be a good idea.



All,

I know this is a very late follow up, but spent some more time looking
at this today and found some additional information that I found quite
interesting. I setup two VMs, one that acts as an iSCSI initiator (
CURRENT ) and another that acts as a target ( 10.2-RELEASE ). Both are
running under ESXi v5.5. There are two block devices on the initiator,
da1 and da2, that I used for resize testing ...

[root@iscsi-i /home/mgrooms]# camcontrol devlist
   at scbus1 target 0 lun 0 (cd0,pass0)
  at scbus2 target 0 lun 0 (pass1,da0)
  at scbus2 target 1 lun 0 (pass2,da1)
 at scbus3 target 0 lun 0 (da2,pass3)

The da1 device is a virtual disk hanging off of a VMware virtual SAS
controller ...

[root@iscsi-i /home/mgrooms]# pciconf
...
mpt0@pci0:3:0:0:class=0x010700 card=0x197615ad chip=0x00541000
rev=0x01 hdr=0x00
 vendor = 'LSI Logic / Symbios Logic'
 device = 'SAS1068 PCI-X Fusion-MPT SAS'
 class  = mass storage
 subclass   = SAS

[root@iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  20971440  da1  GPT  (10G)
 40  209714401  freebsd-ufs  (10G)

The da2 device is an iSCSI LUN mounted from my FreeBSD 10.2 VM running
ctld ...

[root@iscsi-i /home/mgrooms]# iscsictl
Target name  Target portalState
iqn.2015-01.lab.shrew:target0iscsi-t.shrew.lab Connected: da2

[root@iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da2
=>   

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 11/27/2015 12:56 PM, Matthew Grooms wrote:
I thought it would be useful to get more output from the geom layer, 
similar to the camcontrol debug output ...


[root@iscsi-i /home/mgrooms]# sysctl kern.geom.debugflags=0x81

When I resize the iSCSI LUN and run the 'camcontrol readcap da2 -h', I 
see this in the log output ...


[root@iscsi-i /home/mgrooms]# tail -f /var/log/messages
...
Nov 27 12:20:07 iscsi-i kernel: (pass3:iscsi1:0:0:0): Capacity data 
has changed
Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0x80973850, 
0xf80003f4e000, 1, 0)
Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0x8097a6b0, 
0xf80003f42b60, 2, 0)
Nov 27 12:20:07 iscsi-i kernel: 
g_resize_provider_event(0xf800038c6700)

Nov 27 12:20:07 iscsi-i kernel: g_part_resize(da2)
Nov 27 12:20:07 iscsi-i kernel: GEOM_PART: da2 was automatically resized.
Nov 27 12:20:07 iscsi-i kernel: Use `gpart commit da2` to save changes 
or `gpart undo da2` to revert them.

Nov 27 12:20:07 iscsi-i kernel: g_raid_taste(RAID, da2)
Nov 27 12:20:07 iscsi-i kernel: g_attach(0xf80003e64780, 
0xf800038c6700)

Nov 27 12:20:07 iscsi-i kernel: g_detach(0xf80003e64780)
Nov 27 12:20:07 iscsi-i kernel: g_destroy_consumer(0xf80003e64780)
Nov 27 12:20:07 iscsi-i kernel: 
g_destroy_geom(0xf800038c9c00(raid:taste))

Nov 27 12:20:07 iscsi-i kernel: g_label_taste(LABEL, da2)

However, when I resize the VMDK disk and run the 'camcontrol readcap 
da1 -h' command, I see nothing in the log output. So it would appear 
that even though cam is reporting the new capacity in the command line 
output, the this info is not being forwarded to geom in this case. 
Maybe the VMware virtual SAS device ( mpt0 ) isn't reporting some 
special capability bit to cam which causes it to squelch the info?


I'm not sure if this is useful but here is what the device info looks 
like at boot time ...


mpt0:  port 0x4000-0x40ff mem 
0xfd4ec000-0xfd4e,0xfd4f-0xfd4f irq 18 at device 0.0 on pci3

mpt0: MPI Version=1.5.0.0
...
da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0:  Fixed Direct Access SCSI-2 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 20480MB (41943040 512 byte sectors)
da0: quirks=0x40
da1 at mpt0 bus 0 scbus2 target 1 lun 0
da1:  Fixed Direct Access SCSI-2 device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 20480MB (41943040 512 byte sectors)
da1: quirks=0x40
...
da2 at iscsi1 bus 0 scbus3 target 0 lun 0
da2:  Fixed Direct Access SPC-4 SCSI device
da2: Serial Number MYSERIAL   0
da2: 150.000MB/s transfers
da2: Command Queueing enabled
da2: 16384MB (33554432 512 byte sectors)



I spent the day looking over the FreeBSD cam and scsi_da source code. 
After sprinkling a bunch of printf's around to see what code paths were 
being called, It's obvious that Edward was correct in assuming that ESXi 
doesn't return any 'Unit Attention' sense information in response to a 
'Read Capacity' request. This kinda makes sense as ESXi emulates SCSI-2 
disk devices and, as far as I can tell, the 0x2A/0x09 ASC/ASCQ sense 
code that denotes 'Capacity Data Has Changed' wasn't defined until the 
SCSCI-3 spec. It's frustrating that the only way to get the scsci_da 
code to call reprobe() is by receiving a command from the device. Would 
something like this work? ...


1) Register a callback using xpt_register_async( daasync, 
AC_REPROBE_DEVICE, path ) that calls reprobe()
2) Implement a new IOCTL in cam_xpt that camcontrol can call with the 
bus:target:lun as the argument
3) have cam_xpt capture the IOCTL request and call xpt_async( 
AC_REPROBE_DEVICE, path ) as a result


This way users would have the option of manually asking cam to 
communicate the new size to geom. The only option now is one or more 
reboots to gain access to the increased disk capacity. If this sounds 
like a reasonable approach, I'll take a stab at implementing it.


Thanks,

-Matthew
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Resizing a zpool as a VMware ESXi guest ...

[Willem Jan Withagen]

On 27-11-2015 06:59, Matthew Grooms wrote:

On 10/16/2014 3:10 AM, Edward Tomasz Napierała wrote:

On 1010T1529, Matthew Grooms wrote:

All,

I am a long time user and advocate of FreeBSD and manage a several
deployments of FreeBSD in a few data centers. Now that these
environments are almost always virtual, it would make sense that FreeBSD
support for basic features such as dynamic disk resizing. It looks like
most of the parts are intended to work. Kudos to the FreeBSD foundation
for seeing the need and sponsoring dynamic increase of online UFS
filesystems via growfs. Unfortunately, it would appear that there are
still problems in this area, such as ...

a) cam/geom recognizing when a drive's size has increased
b) zpool recognizing when a gpt partition size has increased

For example, if I do an install of FreeBSD 10 on VMware using ZFS, I see
the following ...

root@zpool-test:~ #  gpart show
=>  34  16777149  da0  GPT  (8.0G)
  34  10241  freebsd-boot  (512K)
1058   41943042  freebsd-swap  (2.0G)
 4195362  125818213  freebsd-zfs  (6.0G)

If I increase the VM disk size using VMware to 16G and rescan using
camcontrol, this is what I see ...

"camcontrol rescan" does not force fetching the updated disk size.
AFAIK there is no way to do that.  However, this should happen
automatically, if the "other side" properly sends proper Unit Attention
after resizing.  No idea why this doesn't happen with VMWare.
Reboot obviously clears things up.

[..]


Now I want the claim the additional 14 gigs of space for my zpool ...

root@zpool-test:~ # zpool status
pool: zroot
   state: ONLINE
scan: none requested
config:

  NAME  STATE READ
WRITE CKSUM
  zroot ONLINE 0
0 0
gptid/352086bd-50b5-11e4-95b8-0050569b2a04  ONLINE 0
0 0

root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  5.97G   876M  5.11G14%  1.00x  ONLINE  -

The zpool appears to still only have 5.11G free. Lets reboot and try
again ...

Interesting.  This used to work; actually either of those (autoexpand or
online -e) should do the trick.


root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  14.0G   876M  13.1G 6%  1.00x  ONLINE  -

Now I have 13.1G free. I can add this space to any of my zfs volumes and
it picks the change up immediately. So the question remains, why do I
need to reboot the OS twice to allocate new disk space to a volume?
FreeBSD is first and foremost a server operating system. Servers are
commonly deployed in data centers. Virtual environments are now
commonplace in data centers, not the exception to the rule. VMware still
has the vast majority of the private virutal environment market. I
assume that most would expect things like this to work out of the box.
Did I miss a required step or is this fixed in CURRENT?

Looks like genuine bugs (or rather, one missing feature and one bug).
Filling PRs for those might be a good idea.



All,

I know this is a very late follow up, but spent some more time looking
at this today and found some additional information that I found quite
interesting. I setup two VMs, one that acts as an iSCSI initiator (
CURRENT ) and another that acts as a target ( 10.2-RELEASE ). Both are
running under ESXi v5.5. There are two block devices on the initiator,
da1 and da2, that I used for resize testing ...

[root@iscsi-i /home/mgrooms]# camcontrol devlist
   at scbus1 target 0 lun 0 (cd0,pass0)
  at scbus2 target 0 lun 0 (pass1,da0)
  at scbus2 target 1 lun 0 (pass2,da1)
 at scbus3 target 0 lun 0 (da2,pass3)

The da1 device is a virtual disk hanging off of a VMware virtual SAS
controller ...

[root@iscsi-i /home/mgrooms]# pciconf
...
mpt0@pci0:3:0:0:class=0x010700 card=0x197615ad chip=0x00541000
rev=0x01 hdr=0x00
 vendor = 'LSI Logic / Symbios Logic'
 device = 'SAS1068 PCI-X Fusion-MPT SAS'
 class  = mass storage
 subclass   = SAS

[root@iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  20971440  da1  GPT  (10G)
 40  209714401  freebsd-ufs  (10G)

The da2 device is an iSCSI LUN mounted from my FreeBSD 10.2 VM running
ctld ...

[root@iscsi-i /home/mgrooms]# iscsictl
Target name  Target portalState
iqn.2015-01.lab.shrew:target0iscsi-t.shrew.lab Connected: da2

[root@iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# 

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 10/16/2014 3:10 AM, Edward Tomasz Napierała wrote:

On 1010T1529, Matthew Grooms wrote:

All,

I am a long time user and advocate of FreeBSD and manage a several
deployments of FreeBSD in a few data centers. Now that these
environments are almost always virtual, it would make sense that FreeBSD
support for basic features such as dynamic disk resizing. It looks like
most of the parts are intended to work. Kudos to the FreeBSD foundation
for seeing the need and sponsoring dynamic increase of online UFS
filesystems via growfs. Unfortunately, it would appear that there are
still problems in this area, such as ...

a) cam/geom recognizing when a drive's size has increased
b) zpool recognizing when a gpt partition size has increased

For example, if I do an install of FreeBSD 10 on VMware using ZFS, I see
the following ...

root@zpool-test:~ #  gpart show
=>  34  16777149  da0  GPT  (8.0G)
  34  10241  freebsd-boot  (512K)
1058   41943042  freebsd-swap  (2.0G)
 4195362  125818213  freebsd-zfs  (6.0G)

If I increase the VM disk size using VMware to 16G and rescan using
camcontrol, this is what I see ...

"camcontrol rescan" does not force fetching the updated disk size.
AFAIK there is no way to do that.  However, this should happen
automatically, if the "other side" properly sends proper Unit Attention
after resizing.  No idea why this doesn't happen with VMWare.
Reboot obviously clears things up.

[..]


Now I want the claim the additional 14 gigs of space for my zpool ...

root@zpool-test:~ # zpool status
pool: zroot
   state: ONLINE
scan: none requested
config:

  NAME  STATE READ
WRITE CKSUM
  zroot ONLINE 0 0 0
gptid/352086bd-50b5-11e4-95b8-0050569b2a04  ONLINE 0 0 0

root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  5.97G   876M  5.11G14%  1.00x  ONLINE  -

The zpool appears to still only have 5.11G free. Lets reboot and try
again ...

Interesting.  This used to work; actually either of those (autoexpand or
online -e) should do the trick.


root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  14.0G   876M  13.1G 6%  1.00x  ONLINE  -

Now I have 13.1G free. I can add this space to any of my zfs volumes and
it picks the change up immediately. So the question remains, why do I
need to reboot the OS twice to allocate new disk space to a volume?
FreeBSD is first and foremost a server operating system. Servers are
commonly deployed in data centers. Virtual environments are now
commonplace in data centers, not the exception to the rule. VMware still
has the vast majority of the private virutal environment market. I
assume that most would expect things like this to work out of the box.
Did I miss a required step or is this fixed in CURRENT?

Looks like genuine bugs (or rather, one missing feature and one bug).
Filling PRs for those might be a good idea.



All,

I know this is a very late follow up, but spent some more time looking 
at this today and found some additional information that I found quite 
interesting. I setup two VMs, one that acts as an iSCSI initiator ( 
CURRENT ) and another that acts as a target ( 10.2-RELEASE ). Both are 
running under ESXi v5.5. There are two block devices on the initiator, 
da1 and da2, that I used for resize testing ...


[root@iscsi-i /home/mgrooms]# camcontrol devlist
   at scbus1 target 0 lun 0 (cd0,pass0)
  at scbus2 target 0 lun 0 (pass1,da0)
  at scbus2 target 1 lun 0 (pass2,da1)
 at scbus3 target 0 lun 0 (da2,pass3)

The da1 device is a virtual disk hanging off of a VMware virtual SAS 
controller ...


[root@iscsi-i /home/mgrooms]# pciconf
...
mpt0@pci0:3:0:0:class=0x010700 card=0x197615ad chip=0x00541000 
rev=0x01 hdr=0x00

vendor = 'LSI Logic / Symbios Logic'
device = 'SAS1068 PCI-X Fusion-MPT SAS'
class  = mass storage
subclass   = SAS

[root@iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da1
=>  40  20971440  da1  GPT  (10G)
40  209714401  freebsd-ufs  (10G)

The da2 device is an iSCSI LUN mounted from my FreeBSD 10.2 VM running 
ctld ...


[root@iscsi-i /home/mgrooms]# iscsictl
Target name  Target portalState
iqn.2015-01.lab.shrew:target0iscsi-t.shrew.lab Connected: da2

[root@iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
Device Size: 10 G, Block Length: 512 bytes

[root@iscsi-i /home/mgrooms]# gpart show da2
=>  40  

Re: Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

On 10/16/2014 3:17 AM, Garrett Cooper wrote:



On Oct 16, 2014, at 1:10, Edward Tomasz Napierała
tr...@freebsd.org wrote:



camcontrol rescan does not force fetching the updated disk size.
AFAIK there is no way to do that.  However, this should happen
automatically, if the other side properly sends proper Unit
Attention after resizing.  No idea why this doesn't happen with
VMWare. Reboot obviously clears things up.

[..]


Is open-vm-tools installed?

I ask because if I don't have it installed and the kernel modules
loaded, VMware doesn't notify the guest OS of disks being
added/removed.



VMware tools were not installed at the time. I'll try that, but I doubt 
it will make a difference for resizing.



Also, what disk controller are you using?



The ESXi 5.5 default controller for FreeBSD, LSI Logic Parallel ...

mpt0@pci0:0:16:0:	class=0x01 card=0x197615ad chip=0x00301000 
rev=0x01 hdr=0x00

vendor = 'LSI Logic / Symbios Logic'
device = '53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI'
class  = mass storage
subclass   = SCSI

I'll try it with the LSI Logic SAS controller as well to see if that 
makes a difference.


-Matthew
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Resizing a zpool as a VMware ESXi guest ...

[Edward Tomasz Napierała]

On 1010T1529, Matthew Grooms wrote:
 All,
 
 I am a long time user and advocate of FreeBSD and manage a several 
 deployments of FreeBSD in a few data centers. Now that these 
 environments are almost always virtual, it would make sense that FreeBSD 
 support for basic features such as dynamic disk resizing. It looks like 
 most of the parts are intended to work. Kudos to the FreeBSD foundation 
 for seeing the need and sponsoring dynamic increase of online UFS 
 filesystems via growfs. Unfortunately, it would appear that there are 
 still problems in this area, such as ...
 
 a) cam/geom recognizing when a drive's size has increased
 b) zpool recognizing when a gpt partition size has increased
 
 For example, if I do an install of FreeBSD 10 on VMware using ZFS, I see 
 the following ...
 
 root@zpool-test:~ #  gpart show
 =  34  16777149  da0  GPT  (8.0G)
  34  10241  freebsd-boot  (512K)
1058   41943042  freebsd-swap  (2.0G)
 4195362  125818213  freebsd-zfs  (6.0G)
 
 If I increase the VM disk size using VMware to 16G and rescan using 
 camcontrol, this is what I see ...

camcontrol rescan does not force fetching the updated disk size.
AFAIK there is no way to do that.  However, this should happen
automatically, if the other side properly sends proper Unit Attention
after resizing.  No idea why this doesn't happen with VMWare.
Reboot obviously clears things up.

[..]

 Now I want the claim the additional 14 gigs of space for my zpool ...
 
 root@zpool-test:~ # zpool status
pool: zroot
   state: ONLINE
scan: none requested
 config:
 
  NAME  STATE READ 
 WRITE CKSUM
  zroot ONLINE 0 0 0
gptid/352086bd-50b5-11e4-95b8-0050569b2a04  ONLINE 0 0 0
 
 root@zpool-test:~ # zpool set autoexpand=on zroot
 root@zpool-test:~ # zpool online -e zroot 
 gptid/352086bd-50b5-11e4-95b8-0050569b2a04
 root@zpool-test:~ # zpool list
 NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
 zroot  5.97G   876M  5.11G14%  1.00x  ONLINE  -
 
 The zpool appears to still only have 5.11G free. Lets reboot and try 
 again ...

Interesting.  This used to work; actually either of those (autoexpand or
online -e) should do the trick.

 root@zpool-test:~ # zpool set autoexpand=on zroot
 root@zpool-test:~ # zpool online -e zroot 
 gptid/352086bd-50b5-11e4-95b8-0050569b2a04
 root@zpool-test:~ # zpool list
 NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
 zroot  14.0G   876M  13.1G 6%  1.00x  ONLINE  -
 
 Now I have 13.1G free. I can add this space to any of my zfs volumes and 
 it picks the change up immediately. So the question remains, why do I 
 need to reboot the OS twice to allocate new disk space to a volume? 
 FreeBSD is first and foremost a server operating system. Servers are 
 commonly deployed in data centers. Virtual environments are now 
 commonplace in data centers, not the exception to the rule. VMware still 
 has the vast majority of the private virutal environment market. I 
 assume that most would expect things like this to work out of the box. 
 Did I miss a required step or is this fixed in CURRENT?

Looks like genuine bugs (or rather, one missing feature and one bug).
Filling PRs for those might be a good idea.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Resizing a zpool as a VMware ESXi guest ...

[Garrett Cooper]

 On Oct 16, 2014, at 1:10, Edward Tomasz Napierała tr...@freebsd.org wrote:

 camcontrol rescan does not force fetching the updated disk size.
 AFAIK there is no way to do that.  However, this should happen
 automatically, if the other side properly sends proper Unit Attention
 after resizing.  No idea why this doesn't happen with VMWare.
 Reboot obviously clears things up.
 
 [..]

Is open-vm-tools installed?

I ask because if I don't have it installed and the kernel modules loaded, 
VMware doesn't notify the guest OS of disks being added/removed.

Also, what disk controller are you using?

Cheers.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Resizing a zpool as a VMware ESXi guest ...

[Michael Jung]

On 2014-10-16 04:17, Garrett Cooper wrote:
On Oct 16, 2014, at 1:10, Edward Tomasz Napierała tr...@freebsd.org 
wrote:



camcontrol rescan does not force fetching the updated disk size.
AFAIK there is no way to do that.  However, this should happen
automatically, if the other side properly sends proper Unit 
Attention

after resizing.  No idea why this doesn't happen with VMWare.
Reboot obviously clears things up.

[..]


Is open-vm-tools installed?

I ask because if I don't have it installed and the kernel modules
loaded, VMware doesn't notify the guest OS of disks being
added/removed.

Also, what disk controller are you using?

Cheers.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
freebsd-current-unsubscr...@freebsd.org


I duplicated this behavior.  According to gpart The virtual disk does 
not grow

until the freebsd guest is rebooted.

FreeBSD freebsd10 10.0-RELEASE-p6 FreeBSD 10.0-RELEASE-p6 #0: Tue Jun 24 
07:47:37 UTC 2014

r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC

pkg info -- amd64 open-vm-tools-nox11-1280544_8,1 Open VMware tools for 
FreeBSD VMware guests

ESXi reported -- Running, version:2147483647 (3rd-party/Independent)

ESXi-5.5-1331820(A00) Guest Hardware version 10

789  -  S 0:00.54 /usr/local/bin/vmtoolsd -c 
/usr/local/share/vmware-tools/


Id Refs AddressSize Name

1   12 0x8020 15f03b0  kernel
21 0x81a12000 5209 fdescfs.ko
31 0x81a18000 2198 vmmemctl.ko
41 0x81a1b000 23d8 vmxnet.ko
51 0x81a1e000 2bf0 vmblock.ko
61 0x81a21000 81b4 vmhgfs.ko

--mikej
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Resizing a zpool as a VMware ESXi guest ...

[Matthew Grooms]

All,

I am a long time user and advocate of FreeBSD and manage a several 
deployments of FreeBSD in a few data centers. Now that these 
environments are almost always virtual, it would make sense that FreeBSD 
support for basic features such as dynamic disk resizing. It looks like 
most of the parts are intended to work. Kudos to the FreeBSD foundation 
for seeing the need and sponsoring dynamic increase of online UFS 
filesystems via growfs. Unfortunately, it would appear that there are 
still problems in this area, such as ...


a) cam/geom recognizing when a drive's size has increased
b) zpool recognizing when a gpt partition size has increased

For example, if I do an install of FreeBSD 10 on VMware using ZFS, I see 
the following ...


root@zpool-test:~ #  gpart show
=  34  16777149  da0  GPT  (8.0G)
34  10241  freebsd-boot  (512K)
  1058   41943042  freebsd-swap  (2.0G)
   4195362  125818213  freebsd-zfs  (6.0G)

If I increase the VM disk size using VMware to 16G and rescan using 
camcontrol, this is what I see ...


root@zpool-test:~ # camcontrol rescan all
Re-scan of bus 0 was successful
Re-scan of bus 1 was successful
Re-scan of bus 2 was successful
root@zpool-test:~ # gpart show
=  34  16777149  da0  GPT  (8.0G)
34  10241  freebsd-boot  (512K)
  1058   41943042  freebsd-swap  (2.0G)
   4195362  125818213  freebsd-zfs  (6.0G)

The GPT label still appears to be 8G. If I reboot the VM, it picks up 
the correct size ...


root@zpool-test:~ # gpart show
=  34  16777149  da0  GPT  (16G) [CORRUPT]
34  10241  freebsd-boot  (512K)
  1058   41943042  freebsd-swap  (2.0G)
   4195362  125818213  freebsd-zfs  (6.0G)

Now I have 16G to play with. I'll expand the freebsd-zfs partition to 
claim the additional space ...


root@zpool-test:~ # gpart recover da0
da0 recovered

root@zpool-test:~ # gpart show
=  34  33554365  da0  GPT  (16G)
34  10241  freebsd-boot  (512K)
  1058   41943042  freebsd-swap  (2.0G)
   4195362  125818213  freebsd-zfs  (6.0G)
  16777183  16777216   - free -  (8.0G)

root@zpool-test:~ # gpart resize -i 3 da0

root@zpool-test:~ # gpart show
=  34  33554365  da0  GPT  (16G)
34  10241  freebsd-boot  (512K)
  1058   41943042  freebsd-swap  (2.0G)
   4195362  293590373  freebsd-zfs  (14G)

Now I want the claim the additional 14 gigs of space for my zpool ...

root@zpool-test:~ # zpool status
  pool: zroot
 state: ONLINE
  scan: none requested
config:

NAME  STATE READ 
WRITE CKSUM

zroot ONLINE 0 0 0
  gptid/352086bd-50b5-11e4-95b8-0050569b2a04  ONLINE 0 0 0

root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot 
gptid/352086bd-50b5-11e4-95b8-0050569b2a04

root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  5.97G   876M  5.11G14%  1.00x  ONLINE  -

The zpool appears to still only have 5.11G free. Lets reboot and try 
again ...


root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot 
gptid/352086bd-50b5-11e4-95b8-0050569b2a04

root@zpool-test:~ # zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
zroot  14.0G   876M  13.1G 6%  1.00x  ONLINE  -

Now I have 13.1G free. I can add this space to any of my zfs volumes and 
it picks the change up immediately. So the question remains, why do I 
need to reboot the OS twice to allocate new disk space to a volume? 
FreeBSD is first and foremost a server operating system. Servers are 
commonly deployed in data centers. Virtual environments are now 
commonplace in data centers, not the exception to the rule. VMware still 
has the vast majority of the private virutal environment market. I 
assume that most would expect things like this to work out of the box. 
Did I miss a required step or is this fixed in CURRENT?


Thanks,

-Matthew
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org