Re: [smartos-discuss] zfs_nocacheflush and Enterprise SSDs with Power Loss Protection (Intel S3500)

[also agree with Keith's comments] On Mar 3, 2014, at 2:35 PM, Nick wrote: > I've been testing write performance with SSDs and have a few observations and > questions. > > My focus has been on testing random write performance with 4K blocks, which > is a commonly benchmarked item. Do you act

Re: [smartos-discuss] zfs_nocacheflush and Enterprise SSDs with Power Loss Protection (Intel S3500)

Hi Nick, On Mar 7, 2014, at 12:21 PM, Nick wrote: > On Tue, Mar 4, 2014 at 4:32 PM, Keith Wesolowski > wrote: > It's also been my experience that most applications don't actually open > files O_DSYNC and proceed to do tiny writes to them. I've observed > MySQL, for instance, doing lots of 87-

Re: [smartos-discuss] Network setup

On Mar 6, 2014, at 2:49 PM, tono wrote: > Hello, > > I'm looking for advice about redundant NIC setup, specifically avoiding > single point of failure at the switch. I did see the wiki page on link > aggregations. We use non-stacking switches, which means that all members > would need to connec

Re: [smartos-discuss] Setting up mass storage server

On Mar 6, 2014, at 12:42 PM, Alex Adriaanse wrote: > On Mar 6, 2014, at 10:06 AM, Keith Wesolowski > wrote: >> The way we do this for our Manta storage nodes is to have a single >> RAIDZ2 pool with 3 vdevs (the topology ends up being 3x(9+2) with 2 >> spares and a slog in a 36-disk chassis). T

Re: [smartos-discuss] ZFS continually report data written for MTBF calculations

On Apr 7, 2014, at 10:20 AM, Jakob Borg wrote: > 2014-04-07 18:13 GMT+02:00 Evan Rowley : > > > > I'd like to measure the total amount of data written over time to the > > physical devices that make up zpool zils, l2arcs, and vdevs. Each one of > > these physical devices has a projected Mean T

Re: [smartos-discuss] Memory paging

On Apr 8, 2014, at 12:22 PM, Rajesh Poddar wrote: > I have a system with 64gb of ram that I'm running smartos on. It has 5 > Windows Server 2012 KVM VMs each with 8GB of memory. There is also a smartos > zone with 32GB max_physical_memory that runs a very lightweight file server > that i wrote

Re: [smartos-discuss] How to Know where a drive is located basedon ID

On Apr 11, 2014, at 9:52 PM, Greg Zartman wrote: > I started to ask this question once before, but didn't follow up. > > I have a zpool that looks like this: > > [root@LEI-SmartOS ~]# zpool status > > pool: zones

Re: [smartos-discuss] Is zfs deadm man timer tunable?

> On Jan 21, 2016, at 4:25 AM, Fred Liu wrote: zfs deadman timer is tunable. But if you hit it, you've got problems that tuning the deadman won't help. The tunable is zfs_deadman_synctime_ms, which is milliseconds. For example, on a test machine here: [root@elvis ~]# echo zfs_deadman_s

Re: [smartos-discuss] Is zfs dead man timer tunable?

answer far below... > On Jan 21, 2016, at 8:44 PM, Fred Liu wrote: > > > >> -Original Message- >> From: Richard Elling [mailto:richard.ell...@richardelling.com] >> Sent: 星期五, 一月 22, 2016 12:02 >> To: smartos-discuss@lists.smartos.org >> Subje

[smartos-discuss] Re: [developer] Is it possible to roll back a ZPOOL(which cannot be imported) to its last known good state?

> On Jan 25, 2016, at 12:50 PM, Youzhong Yang wrote: > > Hi all, > > Just wondering if anyone has done similar recovery using txg stuff. Yes. I've seen it successfully done only once in my entire life -- a special case where one node had no significant writes during the dual import period. >

Re: [developer] [smartos-discuss] Is zfs dead man timer tunable?

> On Jan 25, 2016, at 11:22 PM, Fred Liu wrote: > > > Yes. I cleaned the whole folder > > Fred savecore serves two functions: 1. copy the dump from the dump device to a filesystem: vmdump.# (done at boot by the dumpadm SMF service) 2. extract the vmcore.# and unix.# from a vmdump.# U

Re: [smartos-discuss] Update gone bad, the Aftermath

> On Jan 28, 2016, at 1:09 AM, Ian Collins wrote: > > the outsider wrote: >> >> Dear all, >> >> Many thanks to all that helped yesterday and tonight for restoring my server. >> >> Everything works fine now, but I have some unanswered questions. Maybe >> someone can shed some light on this. >

Re: [smartos-discuss] Update gone bad, the Aftermath

> On Jan 31, 2016, at 2:59 PM, Ian Collins wrote: > > Richard Elling wrote: >> >>> On Jan 28, 2016, at 1:09 AM, Ian Collins >> <mailto:i...@ianshome.com>> wrote: >>> >>> It isn't usually a good idea to put both log and cache on th

Re: [smartos-discuss] an interesting survey -- the zpool with most disks you have ever built

> On Mar 3, 2016, at 8:35 PM, Fred Liu wrote: > > Hi, > > Today when I was reading Jeff's new nuclear weapon -- DSSD D5's CUBIC RAID > introduction, > the interesting survey -- the zpool with most disks you have ever built > popped in my brain. We test to 2,000 drives. Beyond 2,000 there are

Re: [zfs] [developer] Re: [smartos-discuss] an interesting survey -- the zpool with most disks you have ever built

> On Mar 6, 2016, at 9:06 PM, Fred Liu wrote: > > > > 2016-03-06 22:49 GMT+08:00 Richard Elling <mailto:richard.ell...@richardelling.com>>: > >> On Mar 3, 2016, at 8:35 PM, Fred Liu > <mailto:fred_...@issi.com>> wrote: >> >> Hi,

Re: [smartos-discuss] zfs_arc_max setting for SmartOS

> On Mar 17, 2016, at 12:15 AM, Matthias Götzke wrote: > > Actually, even though that might be true( that zfs will release the ram) , in > our experience the zfs cache is often freed very slowly and sometimes can > even cause the entire machine to become unresponsive for some time. ZFS experi

Re: [smartos-discuss] zfs_arc_max setting for SmartOS

> On Mar 16, 2016, at 8:21 AM, Jon Dison wrote: > > I’m reasonably certain that the VMs are using mostly swap, at least that’s > what top says. Reserving swap is not the same thing as using swap. It is quite common to see swap reservations, especially for KVM zones. How are you measuring swap

Re: [smartos-discuss] Can not install Build20160317T000621Z HPE Proliant DL380 Gen9, 2 CPU (E5-2650 v3)

> On Mar 21, 2016, at 7:50 AM, Benny Kjellgren > wrote: > > Hi, > > I get this on the console : > > NOTICE: System detected 256 cpus, but only 1 cpu(s) were enabled during > boot. > NOTICE: Use "boot-ncpus" parameter to enable more CPU(s). See eeprom(1M). > WARNING: KCS error: ff > >

Re: [smartos-discuss] Can not install Build20160317T000621Z HPE Proliant DL380 Gen9, 2 CPU (E5-2650 v3)

> On Mar 21, 2016, at 10:49 AM, Benny Kjellgren > wrote: > > Hi Richard > > I can verify that disabling the x2APIC made the NOTICE lines to disappear. > Also notice if setting the SmartArray in HBA mode then "format" command > display no disks. yes. It seems several teams are here now. Which

Re: [smartos-discuss] Can not install Build20160317T000621Z HPE Proliant DL380 Gen9, 2 CPU (E5-2650 v3)

we find that bge2 is toxic. more later today... -- richard > On Mar 22, 2016, at 6:36 AM, Benny Kjellgren > wrote: > > Update : > I trace the reboot to /smartdc/lib/smartos_prompt_config.sh > for iface in $(dladm show-phys -pmo link); do > ifconfig $iface plumb 2>/dev/null > done >

Re: [smartos-discuss] Can not install Build20160317T000621Z HPE Proliant DL380 Gen9, 2 CPU (E5-2650 v3)

> On Mar 22, 2016, at 8:10 PM, 许若辰 wrote: > > I found a interesting material benny > http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04633840&DocLang=en&docLocale=en_US > > >

Re: [smartos-discuss] Can not install Build20160317T000621Z HPE Proliant DL380 Gen9, 2 CPU (E5-2650 v3)

> On Mar 23, 2016, at 6:59 AM, Robert Mustacchi wrote: > > On 3/23/16 2:00 , Benny Kjellgren wrote: >> Thank you >> >> "-B pci-reprog=off" solved my problem with SmartOS not booting up. >> >> And have changed from raidz1 with pair of mirrored disks LV >> to raidz2 with single disk raid 0 LV +

Re: [smartos-discuss] Can not install Build20160317T000621Z HPE Proliant DL380 Gen9, 2 CPU (E5-2650 v3)

> On Mar 23, 2016, at 8:29 AM, Robert Mustacchi wrote: > > On 3/23/16 8:22 , Richard Elling wrote: >> >>> On Mar 23, 2016, at 6:59 AM, Robert Mustacchi wrote: >>> >>> On 3/23/16 2:00 , Benny Kjellgren wrote: >>>> Thank you >>>

Re: [smartos-discuss] Can not install Build20160317T000621Z HPE Proliant DL380 Gen9, 2 CPU (E5-2650 v3)

> On Mar 23, 2016, at 8:29 AM, Robert Mustacchi wrote: > >> On 3/23/16 8:22 , Richard Elling wrote: >> >>> On Mar 23, 2016, at 6:59 AM, Robert Mustacchi wrote: >>> >>> On 3/23/16 2:00 , Benny Kjellgren wrote: >>>> Thank you >>

Re: [smartos-discuss] (U)EFI boot for SmartOS?

> On Apr 12, 2016, at 11:05 AM, Robert Mustacchi wrote: > > On 4/12/16 9:13 , Dirk Steinberg wrote: >> Hi, >> >> as far as I can see, the currently included version of GRUB >> in SmartOS only supports legacy-booting. > > That is correct. > >> Does an EFI-enabled version of GRUB for SmartOS/Il

Re: [smartos-discuss] (U)EFI boot for SmartOS?

> On Apr 12, 2016, at 1:39 PM, Dirk Steinberg wrote: > > The root file system actually resides on a ram disk, which cannot be used for > booting. The RAM disk contains a UFS file system. > > If I do not boot from USB or PXE, I like to put the boot files > (kernel and boot_archive, plus a few

Re: [smartos-discuss] (U)EFI boot for SmartOS?

> On Apr 12, 2016, at 2:41 PM, Dirk Steinberg wrote: > >> >> Am 12.04.2016 um 23:30 schrieb Dirk Steinberg > <mailto:d...@steinbergnet.net>>: >> >> >>> Am 12.04.2016 um 23:09 schrieb Richard Elling >>> >> <mailto:richard.e

Re: [smartos-discuss] preallocated zVOL / improving SSD performance

> On Apr 12, 2016, at 3:21 PM, Dirk Steinberg wrote: > > Hi, > > in order to improve long-term performance on consumer-grade SSDs, > I would like to reserve a certain range of LBA addresses on a freshly > TRIMmed SSD to never be written to. That can be done by slicing the > disk and leaving one

Re: [smartos-discuss] preallocated zVOL / improving SSD performance

> On Apr 13, 2016, at 4:40 PM, Daniel Carosone > wrote: > > Yes, agreed and understood. It is a space reservation that ensures some > number of blocks will never be allocated. > > That's not exactly the same as them never being used, due to CoW updates, but > it's very close. Once the pool i

Re: [smartos-discuss] preallocated zVOL / improving SSD performance

> On Apr 14, 2016, at 1:56 AM, Dirk Steinberg wrote: > >> >> Am 14.04.2016 um 02:39 schrieb Richard Elling >> mailto:richard.ell...@richardelling.com>>: >> >> >>> On Apr 13, 2016, at 4:40 PM, Daniel Carosone >> <mailto:daniel.car

Re: [smartos-discuss] low network bandwidth with 40 vms

> On May 3, 2016, at 10:16 AM, Stefan wrote: > > Dear List, > > on our machines with about 40 vservers we observe low network > throughput. It seems to scale inversely with the number of vms on the > respective node: > > # vms bw (Mbps) > - - > 37 60 > 24

Re: [smartos-discuss] low network bandwidth with 40 vms

> On May 4, 2016, at 8:37 AM, Stefan wrote: > > Dear List, > > I would like to provide you with an update on the issue. We now use an > intel 10GbE card (ixgbe) which reduces the %tim in intrstat but there is > no increase in throughput. Next steps are as Robert suggests, the USE method. ixg

Re: [smartos-discuss] Re: Any possibilities in python OSError?

> On May 16, 2016, at 3:40 AM, Fred Liu wrote: > > It could be the slow read from the nfs path -- "/tool/iqs/run". > We see nfs v4 client not working in LX brand(I have already opened OS-5265). > And nfs v3 client is working but it has obvious lag( about 5-10 > seconds) at the very first try to

Re: [smartos-discuss] zpool replace with a disk which has different physical blocksize

> On Jul 19, 2016, at 1:19 PM, smartos-discuss@lists.smartos.org wrote: > > Hi, > > the new model is a 2 TB Seagate NAS HDD ST2000VN000 and as to the report it > has 4K as physical blocksize: > dkmp_ext.dki_lbsize = 512 > dkmp_ext.dki_pbsize = 4096 Friendly advice: you can force this 512e

Re: [smartos-discuss] Can not override physical sector disks

> On Sep 20, 2016, at 6:48 AM, Humberto Ramirez wrote: > > Is there a definitive approach / guide / manual / wiki as to how to properly > work / replace 4k - 512 - 512e disks? This has been asked before, here and in > some other lists and obviously continues to be a source of problems and > c

Re: [smartos-discuss] smartos ssd disk question

> On Jan 23, 2017, at 9:54 AM, Gernot Straßer > wrote: > > Hi Robert, > > so let me try the other way around then: Do you know a device where the > manufacturer explicitly says so? STEC did. They also said synchronize_cache is a nop. I’ve been working a lot with similar Toshibas recently. I

Re: [smartos-discuss] Expanded pool only seeing 2TB of 3TB disks

> On Feb 2, 2017, at 7:31 AM, Gareth Howell wrote: > > Hi > I’ve just upgraded the two mirrored disks in my zones pool from 1TB disks to > 3TB disks by replacing each in turn and then doing zpool replace. > > I then did > 'pool scrub' > and > 'zpool set autoexpand=on' > > I expected 'zpool ge

Re: [smartos-discuss] disk speeds and ramdisks

This is what we call an "/etc/system virus" Unless you understand each of the tunables and what they impact, do yourself a favor and avoid putting them on your system. In some cases, they are documented in the Solaris kernel tunables guide, in other cases they are documented only in source code.

Re: [smartos-discuss] Zfs dirty data max value question

> On Mar 7, 2017, at 2:59 PM, 강경원 wrote: > > Hello > I saw the zfs_dirty_data_max is 4GB. > If we have several vms and the data write request comes with 4GBps=32gbps, > the io will be throttled although we use enough ssd or nvme? zil data write > also can be about 4GB. > Can I know why defaul

Re: [smartos-discuss] sysinfo script modification question

> On Apr 3, 2017, at 11:37 PM, Dale Ghent wrote: > > > I have SMCI servers that have mangled or all-zero UUIDs as well. very common with supermicro gear. You'll also see an occasional bogus 00010002-0003-0004-0005-000600070008. The sysinfo code in kernel recognizes some of these as bogus and

Re: [smartos-discuss] znapzend (or similar) in a pull configuration

> On May 23, 2017, at 6:53 AM, Chris Ferebee wrote: > > Hi all, > > znapzend is wonderful for snapshots and backups. > > However, for backups of internet-facing zones, I would prefer a "pull" rather > than "push" configuration, such that the backup host initiates the connection > to the live

Re: [smartos-discuss] hardware selection question

> On Aug 6, 2017, at 4:01 PM, Steve wrote: > > I saw a Supermicro with 60 drives in a 4U chassis! They max out at 90 I > believe. We use 60 SAS 2.5" drives in 2u chassis with dual, 2-socket + 2 12G SAS HBA controllers. Next week we'll have them on the exhibit floor at Flash Memory Summit in S

Re: [smartos-discuss] zpool detach mistake

> On Sep 27, 2017, at 3:10 AM, Antoine Jacoutot wrote: > > Hello. > > So I made a mistake today replacing a faulty drive in a RAID10 setup. > i.e. 4 mirrors in a pool > I stupidly detached the drive (c7) which means I lost one mirror. > Surprisingly, > it _seems_ everything is still working fi

Re: [smartos-discuss] Remove ZIL from zones pool?

bonnie++ should not be considered until after you read https://blogs.oracle.com/roch/decoding-bonnie -- richard > On Jan 24, 2018, at 5:33 PM, Sam Nicholson wrote: > > Well, I'm wrong. > > I got to thinking about this and decided to test it. See > github.com/SamCN2/zfs-stats > > Turn

Re: [smartos-discuss] ZIL recommendations

On May 8, 2014, at 7:51 AM, Ibrahim Tachijian via smartos-discuss wrote: > Thanks for that list Keith (http://eng.joyent.com/manufacturing). For > everyone who hasn't read it yet I do recommend you to do so. > I'm guessing joyent has gone through a huge amount of different vendors > before d

Re: [smartos-discuss] ZIL recommendations

On May 9, 2014, at 10:32 AM, Len Weincier via smartos-discuss wrote: >> > I see from the manufacturing docs that you have a single 50GB device >> > (270-022 usually). From reading around it seems to be recommend that >> > the slog be mirrored as it is taking the writes and if one fails there

Re: [smartos-discuss] Working backplane/enclosure for SES in mass non-Solaris/fishwork-ZFS deployment

Speaking of lighting LEDs... On May 9, 2014, at 7:11 PM, Robert Mustacchi via smartos-discuss wrote: > On 5/9/14 18:01 , Fred Liu via smartos-discuss wrote: >> I post here for I don't know where the largest non-Solaris/fishwork-ZFS >> deployment in the world is. But Joyent provide cloud servic

Re: [smartos-discuss] Strange overheat problem with Supermicro 4U chassis

On May 20, 2014, at 2:50 PM, Roodhart, Jeroen via smartos-discuss wrote: > I stand corrected 😊 Of course I assumed it was a firmware v.s. FMA thing, I > should have articulated that more precisely. > These disks (at least the vendor/series) have been the default choice for > these systems fo

Re: [smartos-discuss] DogeOS/SmartOS smf dependencies visualization tool!

On May 29, 2014, at 10:06 PM, Yu Li via smartos-discuss wrote: > Hi, all, > > just let you know that I did some work to visualize the SmartOS smf > dependencies. > > http://www.dogeos.net/smfgraph/ well done! d3 rules! You can adjust the charge and linkDistance to get a better spread. For

Re: [smartos-discuss] How to send mail from SmartOS command line

On Jun 3, 2014, at 10:06 AM, Ganapathy S A via smartos-discuss wrote: > Hi Alain, > > I'm able to ssh/scp to SmartOS (actually a VBox guest in Ubuntu host) but not > the other way around (from SmartOS to Ubuntu)! Following is the error I'm > getting. What config might I miss on the SmartOS si

Re: [smartos-discuss] LTS version of datasets

On Jun 5, 2014, at 1:35 AM, Jonathan Perkin via smartos-discuss wrote: > > It's worth noting at this point that pkgsrc has native support for > reporting on vulnerable packages. We have a pkgsrc security team who > maintain a file containing all known vulnerabilities, and it is > matched agains

Re: [smartos-discuss] SmartOS HA

On Jun 12, 2014, at 1:16 AM, Chris Ferebee via smartos-discuss wrote: > Speaking of VMware, what ever became of Sun AVS a. k. a. SNDR Remote Mirror? > I once set up a HA file server with that years ago, and am told that it’s be > working well to this day. The source is out there somewhere. AF

Re: [smartos-discuss] Joyents experience with HGST drives and performance

On Jul 7, 2014, at 11:28 AM, Keith Wesolowski via smartos-discuss wrote: > On Mon, Jul 07, 2014 at 08:20:43PM +0200, Ibrahim Tachijian wrote: > >> Sounds like a plan Ketih. >> >> And the additional need for a SLOG (like ZeusRam) is def. not required >> because of my pool being set as sync=dis

Re: [smartos-discuss] ghc ./configure generating bad config.status:

On Jul 12, 2014, at 2:21 PM, Alain O'Dea via smartos-discuss wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Fri 11 Jul 2014 12:33:09 PM UTC, Jonathan Perkin via smartos-discuss > wrote: >> * On 2014-07-11 at 13:28 BST, Nahum Shalman via smartos-discuss wrote: >> >>> I've had

Re: [smartos-discuss] 10GbE throughput limited to 1Gb/s, why?

On Jul 19, 2014, at 6:42 AM, Chris Ferebee via smartos-discuss wrote: > > I'm trying to debug a network performance issue. > > I have two servers running SmartOS (20140613T024634Z and 20140501T225642Z), > one is a Supermicro dual Xeon E5649 (64 GB RAM) and the other is a dual Xeon > E5-2620v

Re: [smartos-discuss] Problem with netatalk, very long response times from getcwd()

On Aug 2, 2014, at 6:55 AM, Chris Ferebee via smartos-discuss wrote: > Robert, > > The script you suggested outputs the following, with the server mostly idle > and one OS X client spinning in the Finder, doing its thing to cause the > flood of getcwd() calls. > > # dtrace -n 'fbt::dnlc_rev

Re: [smartos-discuss] illumos / SmartOS equivalent to net.ipv4.ip_nonlocal_bind

On Aug 6, 2014, at 8:51 AM, Keith Wesolowski via smartos-discuss wrote: > On Wed, Aug 06, 2014 at 06:31:24AM -0700, Jon via smartos-discuss wrote: > >> What is the illumos / SmartOS equivalent to Linux's sysctl setting >> net.ipv4.ip_nonlocal_bind ? > > There is none. > > For the benefit of

Re: [smartos-discuss] Strange overheat problem with Supermicro 4U chassis

On May 24, 2014, at 1:54 AM, Ian Collins via smartos-discuss wrote: > Ian Collins via smartos-discuss wrote: >> Ian Collins via smartos-discuss wrote: >>> Richard Elling via smartos-discuss wrote: >>>> Firmware rev 0001, 0002, 0004, and a004 work. Firmware r

Re: [smartos-discuss] Deleting large files blocks all IO

On Sep 26, 2014, at 9:00 AM, Rajesh Poddar via smartos-discuss wrote: > When I delete large files (2-3TB) on my smartos system it often blocks all IO > operations. The only info I found online pointed to dedup as a possible > culprit but I have never enabled dedup on my system. If that data

Re: [smartos-discuss] Very slow KVM/Windows Server disk writes

On Oct 3, 2014, at 2:43 AM, Ian Collins via smartos-discuss wrote: > Micky via smartos-discuss wrote: >> Welcome to the SmartOS. >> You are not alone :) >> >> There's no known fix or workaround to this problem, which is most likely due >> to the KVM implementation in illumous. >> > > There

Re: [smartos-discuss] zfs/zpool versions, send/receive, and Solaris 10

On Nov 14, 2014, at 3:45 PM, John Thurston via smartos-discuss wrote: > I tried to "zfs send" a snapshot from my Solaris 10 system to a SmartOS > system. The result was: >> cannot receive: stream has unsupported feature, feature flags = 24 > > Which I interpret as telling me my Solaris system

Re: [smartos-discuss] Filesystem read/write speed in Centos VM

> On Jan 23, 2015, at 12:45 PM, Greg Zartman via smartos-discuss > wrote: > > A picture says a thousand words, so I thought I'd wrap this discussion up > with a KVM windows server disk performance screenshot. Very good results. sweet! -- richard > > > Greg J. Zartman, P.E. > President,