Re: New vendor branch for FreeBSD on Hyper-V. Please review.
Hi. As ATA driver maintainer I've took a look on that part of the code. I have some comments about what I see. Please keep me CC'd, as I am normally not on this list. 1) You are using machine dependent function do_cpuid() in machine independent ATA code. It will just not build on any architecture except amd64 and i386. You should periodically run `make universe` to be sure that you code does not breaks the build for some rare archs. As minimum that code should be wrapped into respective #ifdef's, but I would prefer it to be moved out somewhere to MD code (like MD code that does general VM detection now), as this doesn't really belong to the ATA driver. 2) You are disabling ad ATA disk driver. Here I have two objections: a) ad driver is obsoleted and not used since FreeBSD 9.0, when CAM-based ATA stack replaces it completely, wrapping only controller driver parts of old ata(4) into CAM SIM, which is now only one of four ATA/SATA SIMs in the system. Your change is effectively null now. b) I believe you are using wrong approach here. I believe instead of completely disabling disk driver you should disable only specific Hyper-V emulated ATA controller device instance. I've never used Hyper-V, but I guess it should support PCI device pass-through. Just imagine what happen if somebody wish to pass-through ATA/SATA controller? You will disable it also. 3) Looking on what I suppose is Fast IDE driver in the file hv_storvsc_drv_freebsd.c, it seems more like cut-down SAS, then IDE. It doesn't support ATA commands, it reports itself to CAM as SAS. The only differences I see are disabling LUNs and limiting queue to 1 slot. Just a smile: it is neither really Fast nor at all IDE. :) 4) In you SAS driver in storvsc_action() you have hardcoded CTS_SCSI_FLAGS_TAG_ENB flag. While I guess it should not cause any problems, it wont allow user to control it if he decide to experiment with disabling it. -- Alexander Motin ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: ATA controller and Hyper-V synthetic storage driver handle the IDE disks devices ( but not for the CD-ROM) Re: Hyper-V on FreeBSD
On 07.05.2013 10:40, Victor Miasnikov wrote: prevent default OS driver from attaching to emulated ATA controller when Hyper-V Disable both primary and secondary ATA controller prevent use CD-ROM device in VM ( guest) Because synthetic storage driver not handle CD-ROM devices ( IMHO, even more: Hyper-V host work with CD-ROM only as PCI bus device) Then may be possibility of blocking/hiding specific ATA channels or devices could be investigated. http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-April/001180.html == . . . On 23.04.2013 17:07, Larry Melia wrote: . . . Removing our ATA patches, on the other hand, results in a huge performance loss. This is because the root file system is managed by the ATA driver, which is emulated under Hyper-V. == Place on IDE disk only \boot ( and bootloader), other on SCSI . May be instead of splitting filesystem it could be better to boot from SCSI completely and forget about ATA? Or there is no option ROM for SCSI boot in Hyper-V? -- Alexander Motin ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: more granular detection and control to disable/enable PCI-ATA devices
Hi. On 04.06.2013 18:30, Larry Melia wrote: With you suggestions, I finally was able to get the override driver working-see https://github.com/FreeBSDonHyper-V/freebsd-snapshot/blob/hyperv-dev-ata-override/sys/dev/hyperv/stordisengage/hv_ata_pci_disengage.c. While it operates wonderfully, allowing our enlightened driver to improve performance, some versions of Hyper-V still use the native CD-ROM driver, because there is no enlightened support for it in the hypervisor. From my limited knowledge of the ATA drivers, it seems likely that the PCI-ATA driver be attached when a CD-ROM is detected, but lower-level drivers disabled (during a probe) when a hard drive is detected. On Hyper-V, therefore, a user would be able to configure a PCI/IDE virtual controller with two devices, the first device a hard disk and the second one a CD-ROM. The CD-ROM would operate via the native driver, whereas the hard disk would use the enlightened driver (to improve performance). Is there an easy way to add more granular detection, disabling the native ATA driver(s) selectively for hard drives, while allowing CD-ROM devices to be natively attached? Any suggestions would be very much appreciated. Unfortunately, CAM subsystem used for both ATA and SCSI stacks in FreeBSD 9.x and above is mostly unaware of NewBus infrastructure used for driver probe and attachment. That is why you can't replace driver for a single disk in the same way as you replaced driver for the whole controller. The highest level present in NewBus is ATA channel. So if disk and CD-ROM are always live on different channels, you can create dummy driver not for the whole controller (atapciX), but for single hardcoded ATA channel (ataX). Another possible way is to make controller driver not dummy, making it mostly duplicating default one, but filtering out unwanted devices. That may look like overkill, but it is not necessary so, because ATA stack is quite modularized, and you probably don't need to implement all ATA functionality such as mode setting, etc. Only thing that should be different in your driver is a reset method -- never reporting ATA disks to upper layers, only ATAPI devices. You may find number of drivers for example in sys/dev/ata/chipsets. The later way is definitely more complicated then just a few lines hack blocking CAM ada driver (ATA disk driver), but it still can be made modular and non-invasive. -- Alexander Motin ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: [Hyper-V][camlock] storvsc driver panics during boot with patches from camlock project
Hi. I took some look and think problems are in scan_for_luns() routine: - After the locking changes scanning normally uses different locks, not the SIM one. That probably caused panic. - But I think that scanning is simply not needed there -- FreeBSD CAM scans every new bus automatically on registration (Even for late registered buses it is done I think at least since FreeBSD 8). I think everything should just work if you remove scan_for_luns() at all. - If you still wish to force scan (due to having information about changed list of devices, etc), then you can make CAM do all the magic for you by calling xpt_rescan(). On 24.10.2013 08:34, Oleg Sidorkin wrote: Hello again. Camlock patches are now committed and -CURRENT on Hyper-V now panics with almost the same stacktrace: FreeBSD 11.0-CURRENT #16 r257016: Wed Oct 23 21:08:44 UTC 2013 olsi@current:/usr/obj/usr/src/sys/HYPERVKERNEL amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 CPU: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz (1309.57-MHz K8-class CPU) Origin = GenuineIntel Id = 0x206a7 Family = 0x6 Model = 0x2a Stepping = 7 .. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 10.000 msec storvsc0 on vmbus0 kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x20 fault code = supervisor read data, page not present instruction pointer = 0x20:0x804f58cc stack pointer = 0x28:0xfe011dd5f5d0 frame pointer = 0x28:0xfe011dd5f600 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= resume, IOPL = 0 current process = 0 (hv_control_1 taskq) [ thread pid 0 tid 100047 ] Stopped at turnstile_broadcast+0x8c: movq0x20(%rbx,%rax,1),%rdx db bt Tracing pid 0 tid 100047 td 0xf8000331e000 turnstile_broadcast() at turnstile_broadcast+0x8c/frame 0xfe011dd5f600 __mtx_unlock_sleep() at __mtx_unlock_sleep+0x60/frame 0xfe011dd5f630 unlock_mtx() at unlock_mtx+0x2a/frame 0xfe011dd5f640 _sleep() at _sleep+0x18e/frame 0xfe011dd5f6c0 cam_periph_runccb() at cam_periph_runccb+0x9e/frame 0xfe011dd5f7f0 storvsc_attach() at storvsc_attach+0x6d4/frame 0xfe011dd5f890 device_attach() at device_attach+0x3a2/frame 0xfe011dd5f8f0 hv_vmbus_child_device_register() at hv_vmbus_child_device_register+0xdb/frame 0xfe011dd5f990 vmbus_channel_process_offer() at vmbus_channel_process_offer+0x133/frame 0xfe011dd5f9d0 work_item_callback() at work_item_callback+0x26/frame 0xfe011dd5f9f0 taskqueue_run_locked() at taskqueue_run_locked+0xe6/frame 0xfe011dd5fa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xa8/frame 0xfe011dd5fa70 fork_exit() at fork_exit+0x9a/frame 0xfe011dd5fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe011dd5fab0 --- trap 0, rip = 0, rsp = 0xfe011dd5fb70, rbp = 0 --- Thanks On Tue, Sep 24, 2013 at 3:04 AM, Abhishek Gupta (LIS) abgu...@microsoft.com wrote: Hi Oleg, Please give us some time. I shall look at it. Thanks for reporting. Regards, Abhishek -Original Message- From: owner-freebsd-virtualizat...@freebsd.org [mailto:owner-freebsd-virtualizat...@freebsd.org] On Behalf Of Oleg Sidorkin Sent: Monday, September 23, 2013 7:21 AM To: freebsd-virtualization@freebsd.org Cc: Alexander Motin Subject: [Hyper-V][camlock] storvsc driver panics during boot with patches from camlock project Hello. I'm running the latest current (amd64) under Hyper-V with hyper-v services enabled. If camlock patches are applied (http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch), I'm hitting the following kernel panic during boot: FreeBSD 10.0-ALPHA2 #5 r255762M: Sun Sep 22 16:48:21 UTC 2013 olsi@current:/usr/obj/usr/src/sys/HYPERVKERNEL amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 CPU: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz (1309.17-MHz K8-class CPU) Origin = GenuineIntel Id = 0x206a7 Family = 0x6 Model = 0x2a Stepping = 7 Timecounter Hyper-V frequency 1000 Hz quality 1000 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add vfs.zfs.prefetch_disable=0 to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 10.000 msec storvsc0 on vmbus0 Netvsc initializing... SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0x20 fault code = supervisor read data, page not present instruction pointer = 0x20:0x804f444c stack pointer = 0x28:0xfe011df38610
Re: Bhyve storage improvements
On 27.03.2015 18:47, John Nielsen wrote: Does anyone have plans (or know about any) to implement virtio-scsi support in bhyve? That API does support TRIM and should retain most or all of the low-overhead virtio goodness. I was thinking about that (not really a plans yet, just some thoughts), but haven't found a good motivation and understanding of whole possible infrastructure. I am not sure it worth to emulate SCSI protocol in addition to already done ATA in ahci-hd and simple block in virtio-blk just to get another, possibly faster then AHCI, block storage with TRIM/UNMAP. Really good SCSI disk emulation in CTL in kernel takes about 20K lines of code. It is pointless to duplicate it, and may be complicated for administration to just interface to it. Indeed I've seen virtio-blk being faster then ahci-hd in some tests, but those tests were highly synthetic. I haven't tested it on real workloads, but I have feeling that real difference may be not that large. If somebody wants to check -- more benchmarks are highly welcome! From the theoretical side I'd like to notice that both ATA and SCSI protocols on guests go through additional ATA/SCSI infrastructure (CAM in FreeBSD), absent in case pure block virtio-blk, so they have some more overhead by definition. Main potential benefit I see from using virtio-scsi is a possibility to pass through to client not a block device, but some real SCSI device. It can be some local DVD writer, or remote iSCSI storage. The last would be especially interesting for large production installations. But the main problem I see here is booting. To make user-level loader boot the kernel from DVD or iSCSI, bhyve has to implement its own SCSI initiator, like small second copy of CAM in user-level. Booting kernel from some other local block storage and then attaching to remote iSCSI storage for data can be much easier, but it is not convenient. It is possible to nt connect to iSCSI directly from user-level, but to make kernel CAM do it, and then make CAM provide both block layer for booting and SCSI layer for virtio-scsi, but I am not sure that it is very good from security point to make host system to see virtual disks. Though may be it could work if CAM could block kernel/GEOM access to them, alike it is done for ZVOLs now, supporting geom and dev modes. Though that complicates CAM and the whole infrastructure. -- Alexander Motin ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Bhyve storage improvements (was: Several bhyve quirks)
Hi, Julian. I had some time to try it out today, but I'm still having issues: I've just made experiment alike to your with making bhyve to work on top of GEOM device instead of preferable dev mode of ZVOL. And I indeed reproduced the problem. But the problem that I see is not related to the block size. The block size is reported to the guest correctly as 4K, and as I can see it works as such at least in FreeBSD guest. The problem is in the way how bhyve inter-operates with block/GEOM devices. bhyve sends requests to the kernel with preadv()/pwritev() calls, specifying scatter/gather lists of buffer addresses provided by the guest. But GEOM code can not handle scatter/gather lists, only sequential buffer, and so single request is split into several. The problem is that splitting happens according to scatter/gather elements, and those elements in general case may not be multiple to the block size, that is fatal for GEOM and any block device. I am not yet sure how to fix this problem. The most straightforward way is to copy the data at some point to collect elements of scatter/gather list into something sequential to pass to GEOM, but that requires additional memory allocation, and the copying is not free. May be some cases could be optimized to work without copying but with some clever page mapping, but that seems absolutely not trivial. -- Alexander Motin ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
[Differential] [Updated] D5473: ATA/ATAPI6 device emulation in bhyve
mav added a comment. It was definitely significant amount of work, but I am not sure what motivation was behind it. Why do we need it after already having much more featured AHCI-based ATA/ATAPI emulation? Are there any significant OS not supporting AHCI? There is significant amount of code duplication between implementations, plus this one obviously requires more work to be complete. REVISION DETAIL https://reviews.freebsd.org/D5473 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: iateaca, grehan, neel, tychon, mav Cc: freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] [Commented On] D5473: ATA/ATAPI6 device emulation in bhyve
mav added a comment. In https://reviews.freebsd.org/D5473#117346, @iateaca wrote: > The motivation is to run older versions of operating systems such as FreeBSD 4 which does not have AHCI drivers. > What do you mean by code duplication ? I think only the ATAPI CDROM logic could be common but the current implementation from AHCI can not be used with the ATA data strcutures. If we want to achive this, I think a redesign of AHCI ATAPI is required too. I personally see quite little sense in supporting so old legacy guests. I agree that there can be "some cases", but I am not sure they worth the time spent and code size growth. Other then legacy guests support this code does not give us anything useful -- legacy ATA will be by definition much slower and less functional then its AHCI counterpart. It will require dozens of emulated register accesses per I/O, comparing to only several for AHCI, and won't support command queuing. Though obviously nice and clean unified implementation would look better. In your patch you are one more time reimplementing some subsets of ATA and ATAPI commands handling, already done much wider for AHCI. It would be much better to have single device emulation code, interfacing with different controller code parts. But it would also take much more time (you are rigth that it would require existing code redesign), for the same little reason, so I am not sure how good is that idea. It needs investigation. REVISION DETAIL https://reviews.freebsd.org/D5473 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: iateaca, grehan, neel, tychon, mav Cc: freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
