Alexander: Performance is quite a vague concept. Relative, even. I don't mean to start some philosophy or anything, but it is true.
To begin with how are you connecting to the gluster volumes ? NFS ? Fuse (native glusterfs) ? What volume set are you using ? Striped ? Distributed ? How is your network set ? Jumbo frames ? >From the details you provided, you are not a first timer. Sounds like you've been doing a lot of research. Did you happen to test the performance with other services, for instance, native NFS,or even ol-n-good FPT ? Is network performance ok ? I was fighting some read and write performance issues a couple of weeks ago on my test servers, and it turns out it was the buffers on my NFS client. Tweaking that, and performance for LARGE FILES COPY saturated 1 Gbps. But in the process I've collected an interesting number of gluster and systcl hacks that seemed to improve performance as well. Use at your own risk, for this affects memory usage on your server: For sysctl.conf: net.core.wmem_max=12582912 net.core.rmem_max=12582912 net.ipv4.tcp_rmem= 10240 87380 12582912 net.ipv4.tcp_wmem= 10240 87380 12582912 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 vm.swappiness=10 vm.dirty_background_ratio=1 net.ipv4.neigh.default.gc_thresh2=2048 net.ipv4.neigh.default.gc_thresh3=4096 net.core.netdev_max_backlog=2500 net.ipv4.tcp_mem= 12582912 12582912 12582912 If using a NFS client, use the following mount options: -o rw,async,vers=3,rsize=65536,wsize=65536 Gliuster options I am currently using: network.remote-dio: on cluster.eager-lock: enable performance.stat-prefetch: off performance.io-cache: off performance.read-ahead: off performance.quick-read: off network.ping-timeout: 20 nfs.nlm: off nfs.addr-namelookup: off Other gluster options I found elsewhere and are worth a try: gluster volume set BigVol diagnostics.brick-log-level WARNING gluster volume set BigVol diagnostics.client-log-level WARNING gluster volume set BigVol nfs.enable-ino32 on gluster volume set BigVol performance.cache-max-file-size 2MB gluster volume set BigVol performance.cache-refresh-timeout 4 gluster volume set BigVol performance.cache-size 256MB gluster volume set BigVol performance.write-behind-window-size 4MB gluster volume set BigVol performance.io-thread-count 32 Now, DO keep in mind: mine is a TEST environment, while yours is a real-life situation. Cheers, Carlos On Mon, Mar 10, 2014 at 7:06 PM, Alexander Valys <[email protected]> wrote: > A quick performance question. > > I have a small cluster of 4 machines, 64 cores in total. I am running a > scientific simulation on them, which writes at between 0.1 and 10 MB/s > (total) to roughly 64 HDF5 files. Each HDF5 file is written by only one > process. The writes are not continuous, but consist of writing roughly 1 > MB of data to each file every few seconds. > > Writing to HDF5 involves a lot of reading the file metadata and random > seeking within the file, since we are actually writing to about 30 > datasets inside each file. I am hosting the output on a distributed > gluster volume (one brick local to each machine) to provide a unified > namespace for the (very rare) case when each process needs to read the > other's files. > > I am seeing somewhat lower performance than I expected, i.e. a factor of > approximately 4 less throughput than each node writing locally to the bare > drives. I expected the write-behind cache to buffer each write, but it > seems that the writes are being quickly flushed across the network > regardless of what write-behind cache size I use (32 MB currently), and the > simulation stalls while waiting for the I/O operation to finish. Anyone > have any suggestions as to what to look at? I am using gluster 3.4.2 on > ubuntu 12.04. I have flush-behind turned on, and have mounted the volume > with direct-io-mode=disable, and have the cache size set to 256M. > > The nodes are connected via a dedicated gigabit ethernet network, carrying > only gluster traffic (no simulation traffic). > > (sorry if this message comes through twice, I sent it yesterday but was > not subscribed) > _______________________________________________ > Gluster-users mailing list > [email protected] > http://supercolony.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
