This is what I came up with: < Corresponds to currently running 4.1.6 > Corresponds to old 3.10.12
+ diff /var/lib/glusterd/vols/export/export.10.0.1.6.bricks-hdds-brick.vol /var/lib/glusterd-20190112/vols/export/export.10.0.1.6.bricks-hdds-brick.vol 3d2 < option shared-brick-count 0 45d43 < option bitrot disable 62d59 < option worm-files-deletable on 93,98d89 < volume export-selinux < type features/selinux < option selinux on < subvolumes export-io-threads < end-volume < 108c99 < subvolumes export-selinux --- > subvolumes export-io-threads 128a120 > option timeout 0 150d141 < option transport.listen-backlog 1024 158a150 > option ping-timeout 42 + diff /var/lib/glusterd/vols/export/export.10.0.1.7.bricks-hdds-brick.vol /var/lib/glusterd-20190112/vols/export/export.10.0.1.7.bricks-hdds-brick.vol 3d2 < option shared-brick-count 1 45d43 < option bitrot disable 62d59 < option worm-files-deletable on 93,98d89 < volume export-selinux < type features/selinux < option selinux on < subvolumes export-io-threads < end-volume < 108c99 < subvolumes export-selinux --- > subvolumes export-io-threads 128a120 > option timeout 0 150d141 < option transport.listen-backlog 1024 158a150 > option ping-timeout 42 + diff /var/lib/glusterd/vols/export/export.tcp-fuse.vol /var/lib/glusterd-20190112/vols/export/export.tcp-fuse.vol 40d39 < option force-migration off 75d73 < option cache-invalidation on + diff /var/lib/glusterd/vols/export/trusted-export.tcp-fuse.vol /var/lib/glusterd-20190112/vols/export/trusted-export.tcp-fuse.vol 44d43 < option force-migration off 79d77 < option cache-invalidation on Any other volume files were the same. HTH, Diego On Tue, Jan 15, 2019 at 2:04 PM Davide Obbi <[email protected]> wrote: > i think you can find the volume options doing a grep -R option > /var/lib/glusterd/vols/ and the .vol files show the options > > On Tue, Jan 15, 2019 at 2:28 PM Diego Remolina <[email protected]> wrote: > >> Hi Davide, >> >> The options information is already provided in prior e-mail, see the >> termbin.con link for the options of the volume after the 4.1.6 upgrade. >> >> The gluster options set on the volume are: >> https://termbin.com/yxtd >> >> This is the other piece: >> >> # gluster v info export >> >> Volume Name: export >> Type: Replicate >> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.1.7:/bricks/hdds/brick >> Brick2: 10.0.1.6:/bricks/hdds/brick >> Options Reconfigured: >> performance.stat-prefetch: on >> performance.cache-min-file-size: 0 >> network.inode-lru-limit: 65536 >> performance.cache-invalidation: on >> features.cache-invalidation: on >> performance.md-cache-timeout: 600 >> features.cache-invalidation-timeout: 600 >> performance.cache-samba-metadata: on >> transport.address-family: inet >> server.allow-insecure: on >> performance.cache-size: 10GB >> cluster.server-quorum-type: server >> nfs.disable: on >> performance.io-thread-count: 64 >> performance.io-cache: on >> cluster.lookup-optimize: on >> cluster.readdir-optimize: on >> server.event-threads: 5 >> client.event-threads: 5 >> performance.cache-max-file-size: 256MB >> diagnostics.client-log-level: INFO >> diagnostics.brick-log-level: INFO >> cluster.server-quorum-ratio: 51% >> >> Now I did create a backup of /var/lib/glusterd so if you tell me how to >> pull information from there to compare I can do it. >> >> I compared the file /var/lib/glusterd/vols/export/info and it is the same >> in both, though entries are in different order. >> >> Diego >> >> >> >> >> On Tue, Jan 15, 2019 at 5:03 AM Davide Obbi <[email protected]> >> wrote: >> >>> >>> >>> On Tue, Jan 15, 2019 at 2:18 AM Diego Remolina <[email protected]> >>> wrote: >>> >>>> Dear all, >>>> >>>> I was running gluster 3.10.12 on a pair of servers and recently >>>> upgraded to 4.1.6. There is a cron job that runs nightly in one machine, >>>> which rsyncs the data on the servers over to another machine for backup >>>> purposes. The rsync operation runs on one of the gluster servers, which >>>> mounts the gluster volume via fuse on /export. >>>> >>>> When using 3.10.12, this process would start at 8:00PM nightly, and >>>> usually end up at around 4:30AM when the servers had been freshly rebooted. >>>> From this point, things would start taking a bit longer and stabilize >>>> ending at around 7-9AM depending on actual file changes and at some point >>>> the servers would start eating up so much ram (up to 30GB) and I would have >>>> to reboot them to bring things back to normal as the file system would >>>> become extremely slow (perhaps the memory leak I have read was present on >>>> 3.10.x). >>>> >>>> After upgrading to 4.1.6 over the weekend, I was shocked to see the >>>> rsync process finish in about 1 hour and 26 minutes. This is compared to 8 >>>> hours 30 mins with the older version. This is a nice speed up, however, I >>>> can only ask myself what has changed so drastically that this process is >>>> now so fast. Have there really been improvements in 4.1.6 that could speed >>>> this up so dramatically? In both of my test cases, there would had not >>>> really been a lot to copy via rsync given the fresh reboots are done on >>>> Saturday after the sync has finished from the day before. >>>> >>>> In general, the servers (which are accessed via samba for windows >>>> clients) are much faster and responsive since the update to 4.1.6. Tonight >>>> I will have the first rsync run which will actually have to copy the day's >>>> changes and will have another point of comparison. >>>> >>>> I am still using fuse mounts for samba, due to prior problems with vsf >>>> =gluster, which are currently present in Samba 4.8.3-4, and already >>>> documented in bugs, for which patches exist, but no official updated samba >>>> packages have been released yet. Since I was going from 3.10.12 to 4.1.6 I >>>> also did not want to change other things to make sure I could track any >>>> issues just related to the change in gluster versions and eliminate other >>>> complexity. >>>> >>>> The file system currently has about 16TB of data in >>>> 5142816 files and 696544 directories >>>> >>>> I've just ran the following code to count files and dirs and it took >>>> 67mins 38.957 secs to complete in this gluster volume: >>>> https://github.com/ChristopherSchultz/fast-file-count >>>> >>>> # time ( /root/sbin/dircnt /export ) >>>> /export contains 5142816 files and 696544 directories >>>> >>>> real 67m38.957s >>>> user 0m6.225s >>>> sys 0m48.939s >>>> >>>> The gluster options set on the volume are: >>>> https://termbin.com/yxtd >>>> >>>> # gluster v status export >>>> Status of volume: export >>>> Gluster process TCP Port RDMA Port >>>> Online Pid >>>> >>>> ------------------------------------------------------------------------------ >>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >>>> 13986 >>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >>>> 9953 >>>> Self-heal Daemon on localhost N/A N/A Y >>>> 21934 >>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>> 4598 >>>> Self-heal Daemon on 10.0.1.6 N/A N/A Y >>>> 14485 >>>> >>>> Task Status of Volume export >>>> >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> Truth, there is a 3rd server here, but no bricks on it. >>>> >>>> Thoughts? >>>> >>>> Diego >>>> >>>> >>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> >>>> Virus-free. >>>> www.avast.com >>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> >>>> <#m_1235711297687490073_m_-657708050556748564_m_-2130281720557425520_m_-4021393732076721680_m_8084651329793795211_m_7462352325940458688_m_-6479459361629161759_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> [email protected] >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> Hi Diego, >>> >>> Besides the actual improvements made in the code i think new releases >>> might implement volume options by default that before might have had >>> different setting. I would have been interesting to diff "gluster volume >>> get <volname> all" befor and after the upgrade. Just for curiosity and i am >>> trying to figure out volume options for rsync kind of workloads can you >>> share the command output anyway along with gluster volume info <volname>? >>> >>> thanks >>> >>> > > -- > Davide Obbi > Senior System Administrator > > Booking.com B.V. > Vijzelstraat 66-80 Amsterdam 1017HL Netherlands > Direct +31207031558 > [image: Booking.com] <https://www.booking.com/> > Empowering people to experience the world since 1996 > 43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 > million reported listings > Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG) >
_______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
