Hi List,
I Searching for advice from somebody, who use Legacy client like ESXi with Ceph
I try to build High-performance fault-tolerant storage with Ceph 0.94
In production i have 50+ TB of VMs (~800 VMs)
8 NFS servers each:
2xIntel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
12xSeagate ST2000NM0023
1xLSI Nytro™ MegaRAID® NMR 8110-4i
96 GB of RAM
4x 1 GBE links in Balance-ALB mode (I don't have problem with network
throughput)
Now in lab. i have build 3 node cluster like:
Kernel 4.2
Intel(R) Xeon(R) CPU 5130 @ 2.00GHz
16 Gb of RAM
6xSeagate ST2000NM0033
2x 1GBE in Balance-ALB
i.e. each node is a MON and 6 OSDs
Config like:
osd journal size = 16384
osd pool default size = 2
osd pool default min size = 2
osd pool default pg num = 256
osd pool default pgp num = 256
osd crush chooseleaf type = 1
filestore max sync interval = 180
For attach RBD Storage to ESXi i create a 2 VMs:
2 cores
2 GB RAM
Kernel 4.3
Each vm map big RBD volume and proxy it by LIO to ESXi
ESXi see VMs like iSCSI Target server in Active/Passive mode
RBD created with --image-shared and --image-format 2 keys
My Questions:
1. I have architecture problem?
2. May be you have ideas?
3. ESXi working with iSCSI storage very slow(30-60 Mb/s read/write),
but this is can be a ESXi problem, later i will test this with more
modern Hypervisor server
4. Proxy VMs working not too bad with storage, but fio shows too low numbers:
[global]
size=128g # File size
filename=/storage/testfile.fio
numjobs=1 # One tread
runtime=600 # 10m for each test
ioengine=libaio # Use async io
# Pseude random data, can be compressed by 15%
buffer_compress_percentage=15
overwrite=1 # Overwrite data in file
end_fsync=1 # Doing fsync, at the and of test, for sync OS buffers
direct=1 # Bypass OS cache
startdelay=30 # Pause between tests
bs=4k # Block size for io requests
iodepth=64 # Count of IO request, what can be requested asynchronously
rw=randrw # Random Read/Write
####################################################
# IOMeter defines the server loads as the following:
# iodepth=1 # Linear
# iodepth=4 # Very Light
# iodepth=8 # Light
# iodepth=64 # Moderate
# iodepth=256 # Heavy
####################################################
[Disk-4k-randomrw-depth-1]
rwmixread=50
iodepth=1
stonewall # Do each test separated
####################################################
[Disk-4k-randomrw-depth-8]
rwmixread=50
iodepth=8
stonewall
####################################################
[Disk-4k-randomrw-depth-64]
rwmixread=50
stonewall
####################################################
[Disk-4k-randomrw-depth-256]
rwmixread=50
iodepth=256
stonewall
####################################################
[Disk-4k-randomrw-depth-512]
rwmixread=50
iodepth=512
stonewall
####################################################
[Disk-4k-randomrw-depth-1024]
rwmixread=50
iodepth=1024
stonewall
-- cut --
RBD-LIO-PROXY:
-- cut --
Disk-4k-randomrw-depth-512: (groupid=4, jobs=1): err= 0: pid=10601:
Sat Nov 7 13:59:49 2015
read : io=770772KB, bw=1282.1KB/s, iops=320, runt=600813msec
clat (msec): min=141, max=8456, avg=715.87, stdev=748.55
write: io=769400KB, bw=1280.7KB/s, iops=320, runt=600813msec
clat (msec): min=158, max=9862, avg=878.73, stdev=905.47
-- cut --
One of node in Raid0:
Disk-4k-randomrw-depth-512: (groupid=4, jobs=1): err= 0: pid=4652: Fri
Oct 30 16:29:00 2015
read : io=258500KB, bw=2128.4KB/s, iops=532, runt=121455msec
clat (msec): min=1, max=3983, avg=484.80, stdev=478.39
write: io=257568KB, bw=2120.8KB/s, iops=530, runt=121455msec
clat (usec): min=217, max=3976.1K, avg=478327.33, stdev=480695.05
-- cut --
By me expirience with ScaleIO, must get on proxy node numbers like ~1000 IOPs
I can provide full FIO config and logs if it needed,
i just try to fix perfomance problem and search for advice
5. May be i must change my FIO config?
6. May be i missing something?
If someone have a expirience with similar solutions, story and links a
welcomed -.-
--
Have a nice day,
Timofey.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
