Here are three profile reports from 60-second intervals:
Ubuntu 18.04 system with low load:
https://pastebin.com/XzgmjeuJ
Ubuntu 14.04 system with low load:
https://pastebin.com/5BEHDFwq
Ubuntu 14.04 system with high load:
https://pastebin.com/CFSWW4qn
Each of these systems is "gluster1" in the report. In each cluster,
there are two bricks, gluster1:/md3/gluster and gluster2:/md3/gluster.
The systems are identical hardware-wise (I noticed this morning that the
18.04 upgrade applied a powersave governor to the CPU. I changed it to
the performance governor before running the profile, but that doesn't
seem to have changed the iowait behavior or the profile report appreciably).
What jumps out at me for the 18.04 systems is:
1) The excessively high average latency of the FINODELK operations on
the *local* brick (i.e. gluster1:/md3/gluster). The latency is far lower
for these FINODELK operations against the other node's brick
(gluster2:/md3/gluster). This is puzzling to me.
2) Almost double higher average latency for FSYNC operations against
both the gluster1 and gluster2 bricks.
On the 14.04 systems, the number of FINODELK operations performed during
the 60-second interval is much lower (even on the highload system). And
the latencies are lower.
Regards,
-Kartik
On 2/21/19 12:18 AM, Amar Tumballi Suryanarayan wrote:
If you have both systems to get some idea, can you get the `gluster
profile info' output? That helps a bit to understand the issue.
On Thu, Feb 21, 2019 at 8:20 AM Kartik Subbarao <[email protected]
<mailto:[email protected]>> wrote:
We're running gluster on two hypervisors running Ubuntu. When we
upgraded from Ubuntu 14.04 to 18.04, it upgraded gluster from
3.4.2 to
3.13.2. As soon as we upgraded and since then, we've been seeing
substantially higher iowait on the system, as measured by top and
iotop,
and iotop indicates that glusterfsd is the culprit. For some reason,
glusterfsd is doing more disk reads and/or those reads are being
held up
up at a greater rate. The guest VMs are also seeing more iowait --
their
images are hosted on the gluster volume. This is causing inconsistent
responsiveness from the services hosted on the VMs.
I'm looking for any recommendations on how to troubleshoot and/or
resolve this problem. We have other sites that are still running
14.04,
so I can compare/contrast any configuration parameters and
performance.
The block scheduler on 14.04 was set to deadline and 18.04 was set to
cfq. But changing the 18.04 scheduler to deadline didn't make any
difference.
I was wondering whether glusterfsd on 18.04 isn't caching as much
as it
should. We tried increasing performance.cache-size substantially but
that didn't make any difference.
Another option we're considering but haven't tried yet is
upgrading to
gluster 5.3 by back-porting the package from Ubuntu 19.04 to
18.04. Does
anyone think this might help?
Is there any particular debug logging we could set up or other
commands
we could run to troubleshoot this better? Any thoughts, suggestions,
ideas would be greatly appreciated.
Thanks,
-Kartik
_______________________________________________
Gluster-users mailing list
[email protected] <mailto:[email protected]>
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Amar Tumballi (amarts)
_______________________________________________
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
