Re: [Gluster-users] How to partition directory structur for 300K files?
Hello, did you do some basic tuning to help anyway ? using latest version ? in replication or only distributed ? Why using NFS and not native fuse client to mount volume? did you install VM tools (if using VMware fusion) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com : I am running into trouble while syncing (rsync, cp ... ) my files to glusterfs. After about 50K files, one machine dies and has to be rebooted. As there are about 300K files in one directory, I am thinking about to cluster that in a directory structure in order to overcome that problem. e.g. /0001/filename /0002/filename That would cut down the amount of files in one directory. However this is something I would like to avoid if possible due to SEO - changing the url of the file brings a lot of trouble. The system underneith are 2 seperate VM instances, each running ubuntu 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 retina with capable SSD and 1G internal network between the VMs. Thank you for any help on this. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] How to partition directory structur for 300K files?
I am running into trouble while syncing (rsync, cp ... ) my files to glusterfs. After about 50K files, one machine dies and has to be rebooted. As there are about 300K files in one directory, I am thinking about to cluster that in a directory structure in order to overcome that problem. e.g. /0001/filename /0002/filename That would cut down the amount of files in one directory. However this is something I would like to avoid if possible due to SEO - changing the url of the file brings a lot of trouble. The system underneith are 2 seperate VM instances, each running ubuntu 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 retina with capable SSD and 1G internal network between the VMs. Thank you for any help on this. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to partition directory structur for 300K files?
Re, putting back mailing list so they keep up. With newer version, fuse perform much better, and provide transparent failover. As both brick are 2 VM in same host, latency will not be an issue. Most important is to ensure you have drivers /tools inside VM to get best perf. on client and servers, I use this in /etc/sysctl.conf vm.swappiness=0 net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem=4096 87380 33554432 net.ipv4.tcp_wmem=4096 65536 33554432 # increase the length of the processor input queue net.core.netdev_max_backlog=3 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp options I set on gluster volumes: server.allow-insecure: on performance.client-io-threads: on performance.read-ahead: on performance.readdir-ahead: enable performance.cache-size: 1GB performance.io-thread-count: 16 Options I set on brick in fstab for XFS mounted volumes used by gluster: defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 16:11 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com : re your questions: did you do some basic tuning to help anyway ? no, this is a basica setup. Can you please direct me to the most important tuning parameters to look at? using latest version ? glusterfs 3.7.3 built on Jul 28 2015 15:14:43 in replication or only distributed ? re 2 Why using NFS and not native fuse client to mount volume? I was reading that the NFS-Client is better with small files. (typical 2-20KB in my case) did you install VM tools (if using VMware fusion) ? I am using virtualbox 5.0.3 on Mac OS X 10.10 2015-08-24 16:01 GMT+02:00 Mathieu Chateau mathieu.chat...@lotp.fr: Hello, did you do some basic tuning to help anyway ? using latest version ? in replication or only distributed ? Why using NFS and not native fuse client to mount volume? did you install VM tools (if using VMware fusion) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com: I am running into trouble while syncing (rsync, cp ... ) my files to glusterfs. After about 50K files, one machine dies and has to be rebooted. As there are about 300K files in one directory, I am thinking about to cluster that in a directory structure in order to overcome that problem. e.g. /0001/filename /0002/filename That would cut down the amount of files in one directory. However this is something I would like to avoid if possible due to SEO - changing the url of the file brings a lot of trouble. The system underneith are 2 seperate VM instances, each running ubuntu 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 retina with capable SSD and 1G internal network between the VMs. Thank you for any help on this. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to partition directory structur for 300K files?
thank you for the recommendation on parameters. I tried: gs2:/volume1/data/nfs glusterfs defaults,_netdev,backupvolfile-server=gs1,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier 0 0 The system reports: #sudo mount -a Invalid option noatime Same with nodiratime and logbufs as soon as I remove noatime 2015-08-24 16:48 GMT+02:00 Mathieu Chateau mathieu.chat...@lotp.fr: Re, putting back mailing list so they keep up. With newer version, fuse perform much better, and provide transparent failover. As both brick are 2 VM in same host, latency will not be an issue. Most important is to ensure you have drivers /tools inside VM to get best perf. on client and servers, I use this in /etc/sysctl.conf vm.swappiness=0 net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem=4096 87380 33554432 net.ipv4.tcp_wmem=4096 65536 33554432 # increase the length of the processor input queue net.core.netdev_max_backlog=3 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp options I set on gluster volumes: server.allow-insecure: on performance.client-io-threads: on performance.read-ahead: on performance.readdir-ahead: enable performance.cache-size: 1GB performance.io-thread-count: 16 Options I set on brick in fstab for XFS mounted volumes used by gluster: defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 16:11 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com: re your questions: did you do some basic tuning to help anyway ? no, this is a basica setup. Can you please direct me to the most important tuning parameters to look at? using latest version ? glusterfs 3.7.3 built on Jul 28 2015 15:14:43 in replication or only distributed ? re 2 Why using NFS and not native fuse client to mount volume? I was reading that the NFS-Client is better with small files. (typical 2-20KB in my case) did you install VM tools (if using VMware fusion) ? I am using virtualbox 5.0.3 on Mac OS X 10.10 2015-08-24 16:01 GMT+02:00 Mathieu Chateau mathieu.chat...@lotp.fr: Hello, did you do some basic tuning to help anyway ? using latest version ? in replication or only distributed ? Why using NFS and not native fuse client to mount volume? did you install VM tools (if using VMware fusion) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com: I am running into trouble while syncing (rsync, cp ... ) my files to glusterfs. After about 50K files, one machine dies and has to be rebooted. As there are about 300K files in one directory, I am thinking about to cluster that in a directory structure in order to overcome that problem. e.g. /0001/filename /0002/filename That would cut down the amount of files in one directory. However this is something I would like to avoid if possible due to SEO - changing the url of the file brings a lot of trouble. The system underneith are 2 seperate VM instances, each running ubuntu 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 retina with capable SSD and 1G internal network between the VMs. Thank you for any help on this. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to partition directory structur for 300K files?
Hello, this is to do on brick, not on client side Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 17:57 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com : thank you for the recommendation on parameters. I tried: gs2:/volume1/data/nfs glusterfs defaults,_netdev,backupvolfile-server=gs1,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier 0 0 The system reports: #sudo mount -a Invalid option noatime Same with nodiratime and logbufs as soon as I remove noatime 2015-08-24 16:48 GMT+02:00 Mathieu Chateau mathieu.chat...@lotp.fr: Re, putting back mailing list so they keep up. With newer version, fuse perform much better, and provide transparent failover. As both brick are 2 VM in same host, latency will not be an issue. Most important is to ensure you have drivers /tools inside VM to get best perf. on client and servers, I use this in /etc/sysctl.conf vm.swappiness=0 net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem=4096 87380 33554432 net.ipv4.tcp_wmem=4096 65536 33554432 # increase the length of the processor input queue net.core.netdev_max_backlog=3 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp options I set on gluster volumes: server.allow-insecure: on performance.client-io-threads: on performance.read-ahead: on performance.readdir-ahead: enable performance.cache-size: 1GB performance.io-thread-count: 16 Options I set on brick in fstab for XFS mounted volumes used by gluster: defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 16:11 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com: re your questions: did you do some basic tuning to help anyway ? no, this is a basica setup. Can you please direct me to the most important tuning parameters to look at? using latest version ? glusterfs 3.7.3 built on Jul 28 2015 15:14:43 in replication or only distributed ? re 2 Why using NFS and not native fuse client to mount volume? I was reading that the NFS-Client is better with small files. (typical 2-20KB in my case) did you install VM tools (if using VMware fusion) ? I am using virtualbox 5.0.3 on Mac OS X 10.10 2015-08-24 16:01 GMT+02:00 Mathieu Chateau mathieu.chat...@lotp.fr: Hello, did you do some basic tuning to help anyway ? using latest version ? in replication or only distributed ? Why using NFS and not native fuse client to mount volume? did you install VM tools (if using VMware fusion) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern merlin.morgenst...@gmail.com: I am running into trouble while syncing (rsync, cp ... ) my files to glusterfs. After about 50K files, one machine dies and has to be rebooted. As there are about 300K files in one directory, I am thinking about to cluster that in a directory structure in order to overcome that problem. e.g. /0001/filename /0002/filename That would cut down the amount of files in one directory. However this is something I would like to avoid if possible due to SEO - changing the url of the file brings a lot of trouble. The system underneith are 2 seperate VM instances, each running ubuntu 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 retina with capable SSD and 1G internal network between the VMs. Thank you for any help on this. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
