Re: [zfs-discuss] drive speeds etc
I have both EVDS and EARS 2TB green drive. And I have to say they are not good to build storage servers. EVDS has compatibility issue with my supermicro appliance. it will hang when doing huge data send or copy. from IOSTAT I can see the data throughput is stuck on green disks with extremely high wait time. for EARS drives, they are ok running opensolaris but veryvery poor performance handling small files. I'm doing SVN via NFS share and it takes triple time CO a repository compare to NEtapp FAS960. but the issue could be resolved by adding SSD as log device. notice that, green disk seek time is 15ms and normal 7200RPM disk is around 8.5ms. I'll try the link provided by Christian to see if it can help the performance. anyway, I've decided to use seagate 7200.11 which is big enough and fast. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] performance leakage when copy huge data
by the way, in HDtune, I saw C7: Ultra DMA CRC error count is a little high which indicates a potential connection issue. Maybe all are caused by the enclosure? Bingo! You are right, I've done a lot of tests and the defect is narrorw down the problem hardware. The two pool works fine in one chassis but after moved to original enclosure, it just failed when CP or ZFS send. I also noticed when the machine bootup reading ZFS configure, there is a warning message. REading ZFS config: *Warning /p...@0,0/pci8086,3...@8/pci15d9,1...@0(mpt0): Discovery in progress, can't verify IO unit config. I did search a lot but cannot find more details. my 2 server configuration: 1. PRoblem chassis supermicro SuperChassis847e2. Tysonberg MB with onboard LSI 1068e (IT mode, which direct expose HD to system without RAID), Single Xeon5520. 2. Good Chassis:Self-developed chassis by other department. S5000WB MB, single E5504, 2 PCIe-4x LSI 3081 HBA card. Seems the SAS cable are all connecting right. I suspect the issue of onboard 1068e and moving the LSI3081 card to the problem server to test. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] performance leakage when copy huge data
Service times here are crap. Disks are malfunctioning in some way. If your source disks can take seconds (or 10+ seconds) to reply, then of course your copy will be slow. Disk is probably having a hard time reading the data or something. Yeah, that should not go over 15ms. I just cannot understand why it starts ok with hundred GB files transfered and then suddenly fall to sleep. by the way, WDIDLE time is already disabled which might cause some issue. I've changed to another system to test ZFS send between 8*1TB pool and 4*1TB pool. hope everythings OK on this case. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] performance leakage when copy huge data
Just to update the status and findings. I've checked TLER settings and they are off by default. I moved the source pool to another chassis and do the 3.8TB send again. this time, not any problems! the difference is 1. New chassis 2. BIGGER memory. 32GB v.s 12GB 3. although wdidle time is disabled by default, I've change the HD mode from silent to performance in HDtune. this is what I once heard from some website that might also fix the disk head park/unpark issue (aka, C1). seems TLER is not the root cause or at least, set to off is ok. my next step will be 1. move back HD to see if it's the performance mode fix the issue 2. if not, add more memory and try again. by the way, in HDtune, I saw C7: Ultra DMA CRC error count is a little high which indicates a potential connection issue. Maybe all are caused by the enclosure? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] performance leakage when copy huge data
Hi all: I'm a new guy who is just started ZFS for half a year. We are using Nexenta in corporate pilot environment. these days, when I was trying to move around 4TB data from an old pool(4*2TB raidz) to new pool (11*2TB raidz2), it seems will never end up successfully. 1. I used CP first. the initial speed is great @ 250MB/s. but it will suddenly become s...@400kb/s when around 400-600GB data is copied. I've tried several times. 2. Use Rsync. Speed is crab, around 15MB/s. it will end up good but performance seems dropped even after Rsync finished. 3. use ZFS send|recv. Speed is close to CP. but still suddenly go down to xxxKB/s after around 1.5TB migrated. 4. I've tried CP to remote server which has 1Gb nic connection. It can be finished but I can see a significant system slow down. what's the root cause of that? is it because the memory is too low? all my ZFS system has 12GB memory. I'm considering adding at least 12GB more to each of them. thanks fei -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] performance leakage when copy huge data
thank you Ian. I've re-build the pool to 9*2TB Raidz2 and start the ZFS send command. result will come out after about 3 hours. thanks fei -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] performance leakage when copy huge data
now it gets extremly slow at around 400G sent. first iostat result is captured when the send operation starts. capacity operationsbandwidth pool alloc free read write read write --- - - - - - - sh001a 37.6G 16.2T 0 1.17K 82 146M raidz2 37.6G 16.2T 0 1.17K 82 146M c0t10d0 - - 0201974 21.0M c0t11d0 - - 0201974 21.1M c0t23d0 - - 0201 1.56K 21.0M c0t24d0 - - 0201 1.26K 21.0M c0t25d0 - - 0201662 21.1M c0t26d0 - - 0201 1.26K 21.1M c0t2d0 - - 0202974 21.1M c0t5d0 - - 0200662 20.9M c0t6d0 - - 0200 1.26K 21.0M --- - - - - - - syspool 10.6G 137G 11 13 668K 137K c3d0s0 10.6G 137G 11 13 668K 137K --- - - - - - - vol1 5.40T 1.85T621 5 76.9M 12.4K raidz1 5.40T 1.85T621 5 76.9M 12.4K c0t22d0 - -280 3 19.5M 14.2K c0t3d0 - -279 3 19.5M 13.9K c0t20d0 - -280 3 19.5M 14.2K c0t21d0 - -280 3 19.5M 13.9K --- - - - - - - --- below result is when ZFS send stuck @ 397G. Seems the HD I/O is quite normal. then, where is the data... notice that, IOstat command response very slow. capacity operationsbandwidth pool alloc free read write read write --- - - - - - - sh001a397G 15.9T 0 1.08K490 136M raidz2 397G 15.9T 0 1.08K490 136M c0t10d0 - - 0185 1.68K 19.4M c0t11d0 - - 0185 1.71K 19.4M c0t23d0 - - 0185 1.99K 19.4M c0t24d0 - - 0185 1.79K 19.4M c0t25d0 - - 0185 2.10K 19.4M c0t26d0 - - 0185 2.07K 19.4M c0t2d0 - - 0185 1.99K 19.4M c0t5d0 - - 0185 2.12K 19.4M c0t6d0 - - 0185 2.23K 19.4M --- - - - - - - syspool 10.6G 137G 2 6 131K 48.0K c3d0s0 10.6G 137G 2 6 131K 48.0K --- - - - - - - vol1 5.40T 1.85T 1009 1 125M 2.85K raidz1 5.40T 1.85T 1009 1 125M 2.85K c0t22d0 - -453 0 31.6M 2.64K c0t3d0 - -452 0 31.6M 2.58K c0t20d0 - -453 0 31.6M 2.64K c0t21d0 - -453 0 31.6M 2.56K --- - - - - - - -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] performance leakage when copy huge data
ve you get dedup enabled? Note the read bandwith is much higher. -- Ian. no, dedup is not enabled since it's still not stable enough even for test environment. here is a JPG of Read/Write indicator. RED line is read and GREEN line is write. you can see, because destination pool has more disks, more throughput ability, so, most of the time, it's just waiting for source pool reading. http://bbs.manmi.com/UploadFile/2010-9/20109911304144848.jpg -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] performance leakage when copy huge data
I dig deeper into it and might find some useful information. I attached an X25 SSD for ZIL to see if it helps. but no luck. I run IOstate -xnz for more details and got interesting result as below.(maybe too long) some explaination: 1. c2d0 is SSD for ZIL 2. c0t3d0, c0t20d0, c0t21d0, c0t22d0 is source pool. 3. you can see the first result is the same as zpool iostate -v, and I've ran -xnz several times, the first result were always good. 4. then, 30s later comes up the second one, and then 3rd, 4th. you can see my source pool disks are extremely slow and busy. One by One. asvc_t is high, %b is high. that's abnormal. 5. I think it might because those 4 drives are the old WD green disks with defect of disk head park/unpark and TLER. other 12 2TB disks are the latest one which already fixed. 6. Let me try to test on 1TB disk and see if any issue. r...@sh1slcs001:/volumes# iostat -xnz 30 extended device statistics r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.40.03.00.3 0.0 0.00.00.2 0 0 c2d0 1.48.9 57.8 40.3 0.0 0.12.55.8 1 5 c3d0 0.00.00.00.0 0.0 0.00.0 986.5 0 0 fd0 2.8 154.8 141.5 16517.7 0.0 2.40.1 15.0 2 25 c0t2d0 408.40.3 29140.40.9 0.0 4.10.1 10.1 2 55 c0t3d0 2.8 154.9 136.7 16519.6 0.0 2.40.1 14.9 2 25 c0t5d0 2.8 155.0 139.4 16520.9 0.0 2.40.1 15.0 2 26 c0t6d0 0.00.00.50.0 0.0 0.00.00.1 0 0 c0t7d0 0.00.00.50.0 0.0 0.00.03.4 0 0 c0t8d0 0.00.00.40.0 0.0 0.00.00.1 0 0 c0t9d0 2.8 154.9 139.2 16523.9 0.0 2.40.1 15.2 2 26 c0t10d0 2.8 155.0 136.1 16525.4 0.0 2.50.1 15.7 2 26 c0t11d0 409.90.3 29152.10.9 0.0 3.90.19.4 2 52 c0t20d0 409.50.3 29152.00.9 0.0 3.90.19.6 2 52 c0t21d0 409.20.3 29155.70.9 0.0 4.10.1 10.0 2 55 c0t22d0 2.8 155.0 138.6 16527.6 0.0 2.40.1 15.2 2 26 c0t23d0 2.9 154.9 138.0 16528.8 0.0 2.40.1 15.1 2 26 c0t24d0 2.9 155.0 136.1 16530.2 0.0 2.50.1 15.8 2 27 c0t25d0 2.8 155.1 139.7 16531.6 0.0 2.40.1 15.5 2 26 c0t26d0 extended device statistics r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.30.01.20.0 0.0 0.00.00.1 0 0 c2d0 0.1 17.70.1 51.7 0.0 0.10.24.1 0 7 c3d0 0.12.10.0 79.8 0.0 0.00.14.0 0 0 c0t2d0 0.20.07.10.0 0.1 2.3 278.5 11365.1 1 46 c0t3d0 0.12.20.0 79.9 0.0 0.00.13.7 0 0 c0t5d0 0.12.30.0 80.0 0.0 0.00.19.2 0 0 c0t6d0 0.12.50.0 80.1 0.0 0.00.13.8 0 0 c0t10d0 0.12.40.0 80.0 0.0 0.00.19.5 0 0 c0t11d0 1.90.0 133.00.0 0.1 2.8 60.2 1520.6 2 51 c0t20d0 1.60.0 110.90.0 0.0 0.00.07.6 0 0 c0t21d0 1.60.0 109.50.0 0.0 0.00.0 11.5 0 0 c0t22d0 0.12.40.0 80.0 0.0 0.00.18.6 0 0 c0t23d0 0.12.60.0 80.0 0.0 0.00.12.9 0 0 c0t24d0 0.12.40.0 79.9 0.0 0.00.16.2 0 0 c0t25d0 0.12.30.0 79.9 0.0 0.00.17.8 0 0 c0t26d0 extended device statistics r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.10.00.60.0 0.0 0.00.00.1 0 0 c2d0 0.0 21.30.0 82.3 0.0 0.10.44.0 0 8 c3d0 0.02.10.0 68.2 0.0 0.00.23.8 0 0 c0t2d0 0.70.0 39.10.0 0.0 0.6 64.0 884.1 1 10 c0t3d0 0.02.20.0 68.2 0.0 0.00.13.5 0 0 c0t5d0 0.02.30.0 68.3 0.0 0.00.13.5 0 0 c0t6d0 0.02.30.0 68.3 0.0 0.00.13.2 0 0 c0t10d0 0.02.20.0 68.3 0.0 0.00.13.7 0 0 c0t11d0 2.00.0 133.70.0 0.0 0.00.09.4 0 0 c0t20d0 2.10.0 135.80.0 0.1 5.2 67.8 2498.1 3 88 c0t21d0 2.80.0 134.40.0 0.0 0.00.03.2 0 0 c0t22d0 0.02.10.0 68.2 0.0 0.00.13.6 0 0 c0t23d0 0.02.40.0 68.4 0.0 0.00.13.1 0 0 c0t24d0 0.02.40.0 68.4 0.0 0.00.13.4 0 0 c0t25d0 0.02.20.0 68.3 0.0 0.00.13.3 0 0 c0t26d0 extended device statistics r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.10.00.60.0 0.0 0.00.00.1 0 0 c2d0 0.1 16.50.1 44.5 0.0 0.10.24.8 0 7 c3d0 0.02.30.0 73.8 0.0 0.00.13.8 0 0 c0t2d0 3.5
