Re: [OmniOS-discuss] Managing KVM virtual machines
For shutdown from host I'm using monitor and system_powerdown qemu command through netcat exactly as you wrote before, and I have this configured in service manifest as well. So on system halt it cleanly shutdown all my vm's with ACPI support, there is no problem with that. I think that it finally solved all my problems with KVM on OmniOS. I can see if vm's are running or not, they are not restarted automatically after shutdown from guest and I'm able to cleanly shutdown them with: svcadm disable vmname. Filip From: Jorge Schrauwen [mailto:sjorge...@blackdot.be] Sent: Monday, September 29, 2014 3:58 PM To: Filip Marvan Cc: omnios-discuss@lists.omniti.com Subject: RE: [OmniOS-discuss] Managing KVM virtual machines This will not cleanly shutdown your vm's at system halt though. But if you can live with that, that will work fine :) smime.p7s Description: S/MIME cryptographic signature ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] Infiniband In Omni
Hey All I was wondering if someone who has been using IB in their lab has any good advice for someone just starting to integrate IB into their ZFS environment. I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have heard its not natively supported and there were issues with the drivers) what i am looking for is this Any Best Practices for using IPoIB with OmniOS? Should i use the Stock drivers build into solaris (are there different ones that work better more stable?) Are there Management tools i can install to run further tests? Any other Guidelines you might recommend. What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards) connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it into the switch and wanted to see if there would be any problems (yes the CX card is 20Gb/s) but this is the best i can offer it based on the drivers. Anything someone might see as an issue (Gonna try iSCSI/NFS with This method direct lines to the hosts) uses CIFS on the 1Gb Ether. Thanks Matt Mabis ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] Ang: Infiniband In Omni
-OmniOS-discuss omnios-discuss-boun...@lists.omniti.com skrev: - Från: Matthew Mabis Sänt av: OmniOS-discuss Datum: 2014-09-29 19:21 Kopia: omnios-discuss@lists.omniti.com omnios-discuss@lists.omniti.com Ärende: [OmniOS-discuss] Infiniband In Omni Hey All I was wondering if someone who has been using IB in their lab has any good advice for someone just starting to integrate IB into their ZFS environment. I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have heard its not natively supported and there were issues with the drivers) what i am looking for is this Any Best Practices for using IPoIB with OmniOS? Should i use the Stock drivers build into solaris (are there different ones that work better more stable?) Are there Management tools i can install to run further tests? Any other Guidelines you might recommend. What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards) connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it into the switch and wanted to see if there would be any problems (yes the CX card is 20Gb/s) but this is the best i can offer it based on the drivers. Anything someone might see as an issue (Gonna try iSCSI/NFS with This method direct lines to the hosts) uses CIFS on the 1Gb Ether. Thanks Matt Mabis Well, I'm interested in that! Why not try srp? Can you run srp initiator from VmWare? Perhaps not ESXi, but from ESX? Have you thought of OpenSM? Rgrds Johan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Infiniband In Omni
IIRC, we never worried much about firmware revisions; they were 2.7.0 or newer. I'd run a subnet manager on a regular linux box, just because it is easier to work with linux daemons than processes on a slow embedded linux on a switch. -nld On Mon, Sep 29, 2014 at 2:31 PM, Matthew Mabis mma...@vmware.com wrote: Thanks Narayan for the info that will really help! I assumed using IPoIB realistically for all the things i wanted. but i might want to test out RDSH and ISER as well. I just wanted to know what things i should expect using this type of card... also what firmwares are you using on your CX1/CX2s? i think all of mine are programmed to 2.7.0 as that was the most stable of firmwares for ESXi using Raphael's (Hypervisor.fr) OpenSM manager :) Matthew the SX6012 Switch that i have has an OpenSM manager on it so i don't have to deal with that specific issue. Here is a link to the switch i got. http://www.mellanox.com/related-docs/prod_ib_switch_systems/SX6012.pdf *Matt Mabis* -- *From:* Matthew Lagoe matthew.la...@subrigo.net *Sent:* Monday, September 29, 2014 11:12 AM *To:* 'Narayan Desai'; Matthew Mabis *Cc:* omnios-discuss@lists.omniti.com *Subject:* RE: [OmniOS-discuss] Infiniband In Omni Also for infiniband you need a subnet manager, if your switch doesn’t have a built in one you have to run like a linux box on the same network just for traffic to work, as there wasn’t any I was able to find for openindiana back in the day, that’s why we went with 10gbe. *From:* OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] *On Behalf Of *Narayan Desai *Sent:* Monday, September 29, 2014 11:03 AM *To:* Matthew Mabis *Cc:* omnios-discuss@lists.omniti.com *Subject:* Re: [OmniOS-discuss] Infiniband In Omni I hate to sound negative, but you're in for a bad time. The current drivers were written by sun/oracle, and only just happen to still work. Parts (like IPoIB) work, but others (EoIB) will randomly panic. Hardware support is limited to connectx 1-2. And if anything goes wrong, there isn't anyone that can really help you. That all said, we had decent success with iscsi/iSER over connectX 1 cards. Fujita Syoyo forward ported the last version of the OFED stack that Sun/Oracle modified up to more recent illumos. Those bits are here: https://github.com/syoyo/solaris-infiniband-tools https://urldefense.proofpoint.com/v1/url?u=https://github.com/syoyo/solaris-infiniband-toolsk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0Am=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0As=5b477f1f1ecb13efc47c34d99b50976160a61dcdc518a7799db1ba2bb2445e05 -nld On Mon, Sep 29, 2014 at 1:20 PM, Matthew Mabis mma...@vmware.com wrote: Hey All I was wondering if someone who has been using IB in their lab has any good advice for someone just starting to integrate IB into their ZFS environment. I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have heard its not natively supported and there were issues with the drivers) what i am looking for is this Any Best Practices for using IPoIB with OmniOS? Should i use the Stock drivers build into solaris (are there different ones that work better more stable?) Are there Management tools i can install to run further tests? Any other Guidelines you might recommend. What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards) connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it into the switch and wanted to see if there would be any problems (yes the CX card is 20Gb/s) but this is the best i can offer it based on the drivers. Anything someone might see as an issue (Gonna try iSCSI/NFS with This method direct lines to the hosts) uses CIFS on the 1Gb Ether. Thanks *Matt Mabis* ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss https://urldefense.proofpoint.com/v1/url?u=http://lists.omniti.com/mailman/listinfo/omnios-discussk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0Am=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0As=f8ffddc7c785147583518256e10cdd31bf7a7f7da0e2b5094c83f3260b713aaa ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS r151012 vioblk not working with ISO
On Sep 29, 2014, at 2:35 PM, Jorge Schrauwen sjorge...@blackdot.be wrote: Just an update, switching to IDE sort of makes it install but it takes about an hour or so and it only boots once after that. It also only enables 1 cpu using either core2due, Nehalem or qemu64 cpu type. Does anybody have a working 012 running inside KVM on 012? I have to ask -- what problem are you solving by running a full KVMed 012 inside 012? What does that buy you that a simple native zone can't? I understand there's virtio issues, but I figured you'd be having them on some other KVM. Don't forget that ZFS-inside-ZFS is akin to running TCP-inside-TCP w.r.t. unusual feedback loops and patterns. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] Debugging reproducable iSCSI initiator hang problem?
We have found a reproducable iSCSI initiator issue in our environment. The short form version of the hang is that if there is a network interruption between one of our OmniOS machines and an iSCSI target and you run 'iscsiadm list target -S' in a roughly 30-second window after the interruption starts and before the OmniOS iSCSI initiator subsystem really notices the problem, 'iscsiadm' will hang in an uninterruptable state. Further iscsiadm commands will also hang and we have seen machines escalate into full-scale system lockups. Because this is completely reproducable for us, we have a 'savecore -L' vmdump file (and can generate as many more as would be useful, and we can fully panic a test system if desired). However I don't have much experience poking around OmniOS crash dumps. Where should I be looking as far as mdb commands and so on go? Is there any particular state that would be most useful to get a test machine into before taking/forcing a crash dump? Thanks in advance. - cks [For people who want to know more about our iSCSI configuration, the setup is described here: http://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFileserverSetupII It may be relevant that we're running multipathing over two separate iSCSI networks; we've not currently tried to reproduce this with only a single network or without multipathing enabled. ] ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS r151012 vioblk not working with ISO
On Sep 29, 2014, at 3:48 PM, Jorge Schrauwen sjorge...@blackdot.be wrote: Build environment, that's the only reason. AFAIK that still does not work in a zone yet. You can build illumos-omnios in a zone, but not *some* of the omnios-build packages (kayak primarily). Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] SATA SSD vs. SAS
Recently purchased a large number of 480GB datacenter grade SSDs with the idea to make a ZPOOL for a VM infrastructure. After receiving, I realized that they are SATA and not SAS, and that I'd completely overlooked that fact. I was planning on using them in a DELL R720XD which only uses two SAS cables for 24x2.5 disks. Since this backplane is obviously a SAS expander/multiplier, it will probably lead to disastrous results with SATA drives. Everything I've read says don't use SAS expanders with SATA drives. (not to mention, that I'll completely saturate the data paths with these). In that vein, I've got a few questions. People choose SATA for SSD, because when it comes to SSD, the price may be 4-10 times as much for SAS. I've seen lots of setups with SAS spinning disks and SATA for L2ARC or ZIL. How are people connecting SATA disks for just these tasks and avoiding problems? Straight onto the SATA port on the motherboard? Are there any SATA only controllers that work well for SSD on Omnios? Does anyone know of any JBODs that are expander/multiplier free that would work for this setup? I have had a hard time finding any. Has anyone had any luck with interposers? I read this report, and it sounded promising till I got to the end. http://xnat.org/blog/xnat-hardware/rsf-1-high-availablity-ssd-pool-for-vm-storage-and-build-space/ Thanks in advance, -- Nate Smith ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS r151012 vioblk not working with ISO
On Sep 29, 2014, at 4:14 PM, Jorge Schrauwen sjorge...@blackdot.be wrote: Excellent news, I would greatly prefer a zone over a kvm! I'll be mostly interested in building illumos-gate and illumos-omnios. You can't build illumos-gate just yet on omnios. But unless you're dinking in specific subsystems, you can develop with illumos-omnios, then shuffle it over to illumos-gate for a last-minute OI build pre-integration. Some more info: Omnios 012 works fine on and is 'usable' on SmartOS using ide mode. Fails the same way with vioblk :( So it seems to be am OmniOS 012 on OmniOS KVM issue. But hey if zones work, i'm not going to waste more time. I build illumos-omnios occasionally on my HDC's omniti zone. make-clobber-to-full in 100 minutes (CPU bound with my low-power Xeon E3). Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS r151012 vioblk not working with ISO
On Sep 29, 2014, at 4:19 PM, Jorge Schrauwen sjorge...@blackdot.be wrote: Alright! I'm stuck home (crushed nerv in foot) the entire week so if you got a bit of time to point me in the right way to setup a zone like that. That would be sweet. Finishing some last bits of testing to with OmniOS vs SmartOS KVM running 012 because I just want to be complete for my self :) Lots of output appended as attachments. These include: - My illumos-omnios.env file for /opt/onbld/bin/nightly. - My list of packages on my omniti zone. I think I'll spin up a nightly now, just to be sure. Dan illumos-omnios.env Description: Binary data pkgs Description: Binary data ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] SATA SSD vs. SAS
Thanks, Fabio, Yeah, the 1015 cards are great and cheap. I used that on my old system for ZIL with the exact same setup you're talking about. Unfortunately, it doesn't scale in the servers I have. I had to shoehorn my zil drives in to bypass the backplane on my previous server. I may have to get another chassis like the one you describe. -Original Message- From: Fábio Rabelo [mailto:fa...@fabiorabelo.wiki.br] Sent: Monday, September 29, 2014 4:25 PM To: Nate Smith; omnios-discuss@lists.omniti.com Subject: Re: [OmniOS-discuss] SATA SSD vs. SAS Hi to all ... I live in Brazil, and here there are things that are VERY hard to find ! So, I need to be creative to get things done ... This as my experiences, till now : I Use Supermicro SATA backplanes ( the ones wirh TQ in the end of the part number ) with a mix of SATA and SAS disks without any kind of issue so far ... The only controller card I used till now are the IBM 1015 flashed to IT , and the breakout cables, one end are sff8087 and the other end have 4 SATA connectors . ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
