On 11/28/2015 6:10 PM, Matthew Grooms wrote:
On 11/27/2015 7:44 PM, Matthew Grooms wrote:
I spent the day looking over the FreeBSD cam and scsi_da source code. After sprinkling a bunch of printf's around to see what code paths were being called, It's obvious that Edward was correct in assuming that ESXi doesn't return any 'Unit Attention' sense information in response to a 'Read Capacity' request. This kinda makes sense as ESXi emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09 ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't defined until the SCSCI-3 spec. It's frustrating that the only way to get the scsci_da code to call reprobe() is by receiving a command from the device. Would something like this work? ...


1) Register a callback using xpt_register_async( daasync, AC_REPROBE_DEVICE, path ) that calls reprobe() 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the bus:target:lun as the argument 3) have cam_xpt capture the IOCTL request and call xpt_async( AC_REPROBE_DEVICE, path ) as a result

This way users would have the option of manually asking cam to communicate the new size to geom. The only option now is one or more reboots to gain access to the increased disk capacity. If this sounds like a reasonable approach, I'll take a stab at implementing it.


Here is a proof of concept patch. I'm a complete noob when it comes to cam, scsi or freebsd kernel development for that matter, so I'm sure it could have been done a better way. In any case, I added a new command to camcontrol that allows you to specify a bus, target and lun as an argument. For example ...

# camcontrol readcap da1 -h
Device Size: 32 G, Block Length: 512 bytes

# gpart show da1
=>      40  58720176  da1  GPT  (28G)
        40  58720176    1  freebsd-ufs  (28G)

Note, I resized the VMDK disk in ESXi. The camcontrol output shows the size as 32G but geom thinks its 28G.

# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00>   at scbus1 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 1.0>          at scbus2 target 0 lun 0 (pass1,da0)
<VMware Virtual disk 1.0>          at scbus2 target 1 lun 0 (pass2,da1)
<FREEBSD CTLDISK 0001>             at scbus3 target 0 lun 0 (da2,pass3)

# camcontrol reprobe 2:1:0

This generates an event that is captured by the scsci da device to forces a reprobe. The kernel output looks almost identical to when the 'Unit Attention' sense data is received ...

Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes or `gpart undo da1` to revert them.

Now that geom knows about the increased disk capacity, I can increase the partition size and grow the fs ...

[root@iscsi-i /home/mgrooms]# gpart show da1
=>      40  67108784  da1  GPT  (32G)
        40  58720176    1  freebsd-ufs  (28G)
  58720216   8388608       - free -  (4.0G)

# gpart resize -i 1 da1
da1p1 resized

# growfs da1p1
Device is mounted read-write; resizing will result in temporary write suspension for /var/data1. It's strongly recommended to make a backup before growing the file system. OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB to 32GB? [Yes/No] Yes
super-block backups (for fsck_ffs -b #) at:
 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672

# df -h
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/da0p3     18G    5.3G     12G    31%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da1p1     31G     32M     28G     0%    /var/data1
/dev/da2p1     15G     32M     14G     0%    /var/data2

Sure would be nice to have something like this in the tree. It's really a drag to have to reboot production VMs to increase disk capacity when it could be easily avoided. I'm not sure what the correct IOCTL should look like. Maybe CAMIOCOMMAND is a better way to go? If someone with some experience with the cam/scsi subsystems was willing to give me some direction I'd be willing to try and rewrite the patch in a way that would be commit worthy. I just need some direction.


Ok, last post until I get some feedback. Here's a new version of the patch complete with man page updates. It communicates via CAMIOCOMMAND instead of introducing a new ioctl value. I tried to model it after the device reset option, hopefully with some degree of success. Functionally it should be the same as the first patch.

Thanks,

-Matthew
Index: sbin/camcontrol/camcontrol.8
===================================================================
--- sbin/camcontrol/camcontrol.8        (revision 291390)
+++ sbin/camcontrol/camcontrol.8        (working copy)
@@ -104,6 +104,9 @@
 .Ic reset
 .Aq all | bus Ns Op :target:lun
 .Nm
+.Ic reprobe
+.Aq bus:target:lun
+.Nm
 .Ic defects
 .Op device id
 .Op generic args
@@ -548,6 +551,9 @@
 connecting to that device.
 Note that this can have a destructive impact
 on the system.
+.It Ic reprobe
+Tell the kernel to re-probe the given bus:target:lun
+(XPT_REPROBE_DEV) by issuing an async event to peripheral device.
 .It Ic defects
 Send the
 .Tn SCSI
Index: sbin/camcontrol/camcontrol.c
===================================================================
--- sbin/camcontrol/camcontrol.c        (revision 291390)
+++ sbin/camcontrol/camcontrol.c        (working copy)
@@ -100,7 +100,8 @@
        CAM_CMD_APM             = 0x00000021,
        CAM_CMD_AAM             = 0x00000022,
        CAM_CMD_ATTRIB          = 0x00000023,
-       CAM_CMD_OPCODES         = 0x00000024
+       CAM_CMD_OPCODES         = 0x00000024,
+       CAM_CMD_REPROBE         = 0x00000025
 } cam_cmdmask;
 
 typedef enum {
@@ -193,6 +194,7 @@
 #endif /* MINIMALISTIC */
        {"rescan", CAM_CMD_RESCAN, CAM_ARG_NONE, NULL},
        {"reset", CAM_CMD_RESET, CAM_ARG_NONE, NULL},
+       {"reprobe", CAM_CMD_REPROBE, CAM_ARG_NONE, NULL},
 #ifndef MINIMALISTIC
        {"cmd", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts},
        {"command", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts},
@@ -273,6 +275,7 @@
 static int rescan_or_reset_bus(path_id_t bus, int rescan);
 static int scanlun_or_reset_dev(path_id_t bus, target_id_t target,
     lun_id_t lun, int scan);
+static int reprobe_dev(path_id_t bus, target_id_t target, lun_id_t lun);
 #ifndef MINIMALISTIC
 static int readdefects(struct cam_device *device, int argc, char **argv,
                       char *combinedopt, int retry_count, int timeout);
@@ -3127,6 +3130,34 @@
 }
 
 static int
+doreprobe(int argc, char **argv)
+{
+       static const char must[] =
+               "you must specify a bus:target:lun to reprobe";
+       int rv;
+       path_id_t bus = CAM_BUS_WILDCARD;
+       target_id_t target = CAM_TARGET_WILDCARD;
+       lun_id_t lun = CAM_LUN_WILDCARD;
+       char *tstr;
+
+       if (argc < 3) {
+               warnx(must);
+               return(1);
+       }
+
+       tstr = argv[optind];
+       while (isspace(*tstr) && (*tstr != '\0'))
+               tstr++;
+       rv = parse_btl(argv[optind], &bus, &target, &lun, &arglist);
+       if (rv != 3) {
+               warnx(must);
+               return(1);
+       }
+
+       return(reprobe_dev(bus, target, lun));
+}
+
+static int
 rescan_or_reset_bus(path_id_t bus, int rescan)
 {
        union ccb ccb, matchccb;
@@ -3382,6 +3413,59 @@
        }
 }
 
+static int
+reprobe_dev(path_id_t bus, target_id_t target, lun_id_t lun)
+{
+       union ccb ccb;
+       struct cam_device *device;
+       int fd;
+
+       device = NULL;
+
+       if (bus == CAM_BUS_WILDCARD) {
+               warnx("invalid bus number %d", bus);
+               return(1);
+       }
+
+       if (target == CAM_TARGET_WILDCARD) {
+               warnx("invalid target number %d", target);
+               return(1);
+       }
+
+       if (lun == CAM_LUN_WILDCARD) {
+               warnx("invalid lun number %jx", (uintmax_t)lun);
+               return(1);
+       }
+
+       fd = -1;
+
+       bzero(&ccb, sizeof(union ccb));
+
+       if ((fd = open(XPT_DEVICE, O_RDWR)) < 0) {
+               warnx("error opening transport layer device %s\n",
+                   XPT_DEVICE);
+               warn("%s", XPT_DEVICE);
+               return(1);
+       }
+
+       ccb.ccb_h.func_code = XPT_REPROBE_DEV;
+       ccb.ccb_h.path_id = bus;
+       ccb.ccb_h.target_id = target;
+       ccb.ccb_h.target_lun = lun;
+
+       if (ioctl(fd, CAMIOCOMMAND, &ccb) < 0) {
+               warn("CAMIOCOMMAND ioctl failed");
+               close(fd);
+               return(1);
+       }
+       close(fd);
+
+       fprintf(stdout, "Re-probe of %d:%d:%jx was successful\n",
+           bus, target, (uintmax_t)lun);
+
+       return(0);
+}
+
 #ifndef MINIMALISTIC
 
 static struct scsi_nv defect_list_type_map[] = {
@@ -8685,6 +8769,7 @@
 #endif /* MINIMALISTIC */
 "        camcontrol rescan     <all | bus[:target:lun]>\n"
 "        camcontrol reset      <all | bus[:target:lun]>\n"
+"        camcontrol reprobe    bus:target:lun\n"
 #ifndef MINIMALISTIC
 "        camcontrol defects    [dev_id][generic args] <-f format> [-P][-G]\n"
 "                              [-q][-s][-S offset][-X]\n"
@@ -9053,6 +9138,7 @@
         */
        if ((cmdlist == CAM_CMD_RESCAN)
         || (cmdlist == CAM_CMD_RESET)
+        || (cmdlist == CAM_CMD_REPROBE)
         || (cmdlist == CAM_CMD_DEVTREE)
         || (cmdlist == CAM_CMD_USAGE)
         || (cmdlist == CAM_CMD_DEBUG))
@@ -9204,6 +9290,9 @@
                case CAM_CMD_RESET:
                        error = dorescan_or_reset(argc, argv, 0);
                        break;
+               case CAM_CMD_REPROBE:
+                       error = doreprobe(argc, argv);
+                       break;
 #ifndef MINIMALISTIC
                case CAM_CMD_READ_DEFECTS:
                        error = readdefects(cam_dev, argc, argv, combinedopt,
Index: sys/amd64/conf/GENERIC
===================================================================
--- sys/amd64/conf/GENERIC      (revision 291390)
+++ sys/amd64/conf/GENERIC      (working copy)
@@ -366,3 +366,6 @@
 
 # The crypto framework is required by IPSEC
 device         crypto                  # Required by IPSEC
+
+# Cam Debug Support
+options                CAMDEBUG
Index: sys/cam/cam_ccb.h
===================================================================
--- sys/cam/cam_ccb.h   (revision 291390)
+++ sys/cam/cam_ccb.h   (working copy)
@@ -193,6 +193,11 @@
                                 * Set SIM specific knob values.
                                 */
 
+       XPT_REPROBE_DEV         = 0x1a,
+                               /*
+                                * Re-probe the sepcified SCSI device.
+                                */
+
        XPT_SMP_IO              = 0x1b | XPT_FC_DEV_QUEUED,
                                /* Serial Management Protocol */
 
@@ -767,6 +772,7 @@
  * Definitions for the asynchronous callback CCB fields.
  */
 typedef enum {
+       AC_UNIT_REPROBE         = 0x8000,/* Device reprobe user request */
        AC_UNIT_ATTENTION       = 0x4000,/* Device reported UNIT ATTENTION */
        AC_ADVINFO_CHANGED      = 0x2000,/* Advance info might have changes */
        AC_CONTRACT             = 0x1000,/* A contractual callback */
Index: sys/cam/cam_xpt.c
===================================================================
--- sys/cam/cam_xpt.c   (revision 291390)
+++ sys/cam/cam_xpt.c   (working copy)
@@ -433,6 +433,13 @@
                                return (EINVAL);
                        }
                        break;
+               case XPT_REPROBE_DEV:
+                       if (inccb->ccb_h.target_id == CAM_TARGET_WILDCARD ||
+                           inccb->ccb_h.target_lun == CAM_LUN_WILDCARD) {
+                               xpt_release_bus(bus);
+                               return (EINVAL);
+                       }
+                       break;
                default:
                        break;
                }
@@ -472,6 +479,29 @@
                        xpt_free_ccb(ccb);
                        break;
 
+               case XPT_REPROBE_DEV: {
+                       struct cam_path path;
+
+                       /*
+                        * Compile a path using the bus, target, and lun the
+                        * user passed in.
+                        */
+                       if (xpt_compile_path(&path, NULL,
+                                           inccb->ccb_h.path_id,
+                                           inccb->ccb_h.target_id,
+                                           inccb->ccb_h.target_lun) !=
+                                           CAM_REQ_CMP){
+                               error = EINVAL;
+                               break;
+                       }
+
+                       /*
+                        * Notify interested parties
+                        */
+                       xpt_async(AC_UNIT_REPROBE, &path, NULL);
+                       break;
+
+               }
                case XPT_DEBUG: {
                        union ccb ccb;
 
@@ -717,7 +747,7 @@
                }
                xpt_unlock_buses();
                break;
-               }
+       }
        default:
                error = ENOTTY;
                break;
Index: sys/cam/scsi/scsi_da.c
===================================================================
--- sys/cam/scsi/scsi_da.c      (revision 291390)
+++ sys/cam/scsi/scsi_da.c      (working copy)
@@ -1702,6 +1702,17 @@
                }
                break;
        }
+       case AC_UNIT_REPROBE:
+       {
+               softc = (struct da_softc *)periph->softc;
+
+               xpt_print(periph->path,
+                   "Re-probe requested\n");
+               softc->flags &= ~DA_FLAG_PROBED;
+               dareprobe(periph);
+
+               break;
+       }
        case AC_UNIT_ATTENTION:
        {
                union ccb *ccb;
@@ -2240,7 +2251,7 @@
         * would be to not attach the device on failure.
         */
        xpt_register_async(AC_SENT_BDR | AC_BUS_RESET | AC_LOST_DEVICE |
-           AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION,
+           AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION | 
AC_UNIT_REPROBE,
            daasync, periph, periph->path);
 
        /*
@@ -3244,12 +3255,11 @@
                                        if (have_sense)
                                                scsi_sense_print(
                                                        &done_ccb->csio);
-                                       else {
+                                       else
                                                xpt_print(periph->path,
                                                    "got CAM status %#x\n",
                                                    done_ccb->ccb_h.status);
-                                       }
-
+                                       
                                        xpt_print(periph->path, "fatal error, "
                                            "failed to attach to device\n");
 
_______________________________________________
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Reply via email to