Hi,
On CentOS 7.5 (3.10.0-957.21.2.el7.x86_64), I am seeing better write
performance when commitlog is using "ext4" versus "xfs" filesystem. It is the
same local/ephemeral disk within an Azure VM and the only change I am making is
switching the filesystem on the disk from "ext4" to "xfs".
Is this expected? Any ideas on how to troubleshoot and tune the filesystem (or
Linux or Cassandra) so that xfs performs same or better than ext4?
Details below.
OpenJDK 1.8.0 (same with Oracle JDK 1.8.0)
Cassandra version: 3.11.4
cassandra.yaml has this default config:
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
I am testing write performance using the following cassandra-stress command:
cqlsh --execute "DROP KEYSPACE IF EXISTS keyspace1;" 10.0.0.4
cassandra-stress write no-warmup n=2000000 cl=LOCAL_QUORUM \
-rate threads=32 throttle=0/s \
-pop seq="1..10000000" -col size=FIXED\(818\) \
-node "10.0.0.4" \
-schema keyspace=keyspace1 \
-log interval=1 -mode native cql3 connectionsPerHost=128
ext4 mount and ops/s look like this:
/dev/sdb1 on /mnt/resource type ext4 (rw,relatime,seclabel,data=ordered)
Connected to cluster: avcassandra, max pending requests per connection 128, max
connections per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type total ops, op/s, pk/s, row/s, mean, med, .95,
.99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv
ms, mb
total, 749, 749, 749, 749, 14.4, 8.4, 43.5,
79.7, 107.0, 131.9, 1.0, 0.00000, 0, 0, 0, 0,
0, 0
total, 4776, 4027, 4027, 4027, 7.9, 4.9, 24.6,
44.8, 65.4, 76.7, 2.0, 0.47357, 0, 0, 0, 0,
0, 0
total, 11129, 6353, 6353, 6353, 5.0, 2.5, 16.4,
26.6, 40.8, 46.5, 3.0, 0.34745, 0, 0, 0, 0,
0, 0
total, 17547, 6418, 6418, 6418, 4.9, 2.2, 16.4,
31.8, 74.1, 83.6, 4.0, 0.25930, 0, 0, 0, 0,
0, 0
total, 28396, 10849, 10849, 10849, 3.0, 1.5, 11.4,
21.2, 37.9, 49.0, 5.0, 0.25513, 0, 0, 0, 0,
0, 0
total, 40808, 12412, 12412, 12412, 2.5, 1.5, 9.0,
19.5, 33.8, 68.0, 6.0, 0.23341, 0, 0, 0, 0,
0, 0
total, 53080, 12272, 12272, 12272, 2.6, 1.3, 10.0,
19.0, 60.8, 73.5, 7.0, 0.20377, 0, 0, 0, 0,
0, 0
total, 67025, 13945, 13945, 13945, 2.3, 1.3, 9.1,
17.7, 30.5, 40.8, 8.0, 0.18276, 0, 0, 0, 0,
0, 0
total, 86886, 19861, 19861, 19861, 1.6, 1.2, 2.9,
9.3, 42.7, 56.5, 9.0, 0.18765, 0, 0, 0, 0,
0, 0
total, 106184, 19298, 19298, 19298, 1.6, 1.2, 3.1,
10.8, 57.8, 61.5, 10.0, 0.17614, 0, 0, 0, 0,
0, 0
total, 128877, 22693, 22693, 22693, 1.4, 1.2, 2.6,
4.1, 14.7, 30.2, 11.0, 0.16866, 0, 0, 0, 0,
0, 0
total, 148331, 19454, 19454, 19454, 1.6, 1.3, 3.0,
5.7, 45.3, 47.7, 12.0, 0.15466, 0, 0, 0, 0,
0, 0
total, 156974, 8643, 8643, 8643, 1.6, 1.2, 3.3,
10.2, 24.6, 41.2, 13.0, 0.14813, 0, 0, 0, 0,
0, 0
total, 170654, 13680, 13680, 13680, 3.6, 1.3, 3.1,
9.2, 890.8, 896.5, 14.0, 0.39489, 0, 0, 0, 0,
0, 0
total, 187715, 17061, 17061, 17061, 1.9, 1.2, 4.5,
13.0, 27.2, 44.2, 15.0, 0.37273, 0, 0, 0, 0,
0, 0
total, 205718, 18003, 18003, 18003, 1.7, 1.3, 3.7,
11.3, 32.8, 53.4, 16.0, 0.35161, 0, 0, 0, 0,
0, 0
total, 212945, 7227, 7227, 7227, 1.8, 1.2, 3.2,
11.2, 71.3, 88.5, 17.0, 0.34545, 0, 0, 0, 0,
0, 0
total, 220825, 7880, 7880, 7880, 6.4, 1.2, 3.1,
11.1, 1200.6, 1209.0, 18.0, 0.42966, 0, 0, 0, 0,
0, 0
total, 241273, 20448, 20448, 20448, 1.5, 1.2, 3.1,
7.3, 23.0, 60.4, 19.0, 0.40131, 0, 0, 0, 0,
0, 0
total, 261889, 20616, 20616, 20616, 1.5, 1.2, 2.9,
8.0, 25.0, 33.5, 20.0, 0.37691, 0, 0, 0, 0,
0, 0
total, 282464, 20575, 20575, 20575, 1.5, 1.2, 2.7,
5.2, 68.6, 72.7, 21.0, 0.35456, 0, 0, 0, 0,
0, 0
total, 305429, 22965, 22965, 22965, 1.4, 1.2, 2.5,
3.8, 17.2, 23.8, 22.0, 0.33370, 0, 0, 0, 0,
0, 0
total, 326504, 21075, 21075, 21075, 1.5, 1.2, 2.9,
5.6, 17.9, 31.5, 23.0, 0.31637, 0, 0, 0, 0,
0, 0
total, 342653, 16149, 16149, 16149, 1.9, 1.3, 3.8,
13.2, 107.0, 124.4, 24.0, 0.30293, 0, 0, 0, 0,
0, 0
total, 360972, 18319, 18319, 18319, 1.7, 1.2, 3.8,
11.5, 36.1, 63.7, 25.0, 0.28988, 0, 0, 0, 0,
0, 0
total, 379092, 18120, 18120, 18120, 1.7, 1.2, 2.9,
8.2, 133.5, 161.1, 26.0, 0.27680, 0, 0, 0, 0,
0, 0
total, 400379, 21287, 21287, 21287, 1.5, 1.2, 2.9,
6.4, 30.8, 43.2, 27.0, 0.26454, 0, 0, 0, 0,
0, 0
total, 422330, 21951, 21951, 21951, 1.4, 1.2, 2.6,
4.3, 21.9, 35.3, 28.0, 0.25311, 0, 0, 0, 0,
0, 0
total, 442491, 20161, 20161, 20161, 1.4, 1.2, 2.7,
5.0, 21.6, 41.5, 29.0, 0.24331, 0, 0, 0, 0,
0, 0
total, 456035, 13544, 13544, 13544, 11.9, 1.3, 3.4,
9.7, 4366.3, 4366.3, 34.0, 0.25639, 0, 0, 0, 0,
0, 0
total, 458788, 2753, 2753, 2753, 1.5, 1.3, 2.9,
4.2, 32.8, 40.9, 35.0, 0.25612, 0, 0, 0, 0,
0, 0
total, 468346, 9558, 9558, 9558, 46.4, 1.4, 7.1,
15.6, 13203.7, 13228.8, 48.0, 0.27131, 0, 0, 0, 0,
0, 0
total, 489463, 21117, 21117, 21117, 1.5, 1.2, 2.8,
8.5, 19.7, 32.8, 49.0, 0.26143, 0, 0, 0, 0,
0, 0
total, 510484, 21021, 21021, 21021, 1.5, 1.2, 2.9,
5.7, 20.2, 28.0, 50.0, 0.25232, 0, 0, 0, 0,
0, 0
total, 532895, 22411, 22411, 22411, 1.4, 1.2, 2.5,
4.8, 28.4, 40.7, 51.0, 0.24335, 0, 0, 0, 0,
0, 0
total, 553656, 20761, 20761, 20761, 1.5, 1.2, 3.0,
9.0, 18.2, 33.0, 52.0, 0.23551, 0, 0, 0, 0,
0, 0
xfs mount and ops/s look like this:
/dev/sdb1 on /mnt/resource type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
Connected to cluster: avcassandra, max pending requests per connection 128, max
connections per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type total ops, op/s, pk/s, row/s, mean, med, .95,
.99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv
ms, mb
total, 1550, 1550, 1550, 1550, 14.9, 12.2, 37.8,
61.5, 75.6, 82.2, 1.0, 0.00000, 0, 0, 0, 0,
0, 0
total, 5742, 4192, 4192, 4192, 7.6, 4.6, 23.4,
36.7, 63.0, 78.8, 2.0, 0.32434, 0, 0, 0, 0,
0, 0
total, 10026, 4284, 4284, 4284, 7.4, 5.1, 20.4,
33.5, 95.2, 112.1, 3.0, 0.22200, 0, 0, 0, 0,
0, 0
total, 15802, 5776, 5776, 5776, 5.5, 3.2, 17.0,
32.8, 50.5, 62.6, 4.0, 0.18920, 0, 0, 0, 0,
0, 0
total, 23728, 7926, 7926, 7926, 4.0, 2.6, 11.6,
23.2, 54.7, 90.6, 5.0, 0.19722, 0, 0, 0, 0,
0, 0
total, 32143, 8415, 8415, 8415, 3.7, 2.5, 11.8,
20.4, 40.8, 48.1, 6.0, 0.17557, 0, 0, 0, 0,
0, 0
total, 41506, 9363, 9363, 9363, 3.4, 2.0, 11.0,
26.1, 72.2, 84.2, 7.0, 0.16341, 0, 0, 0, 0,
0, 0
total, 52696, 11190, 11190, 11190, 2.8, 2.6, 5.0,
10.8, 20.1, 32.3, 8.0, 0.15527, 0, 0, 0, 0,
0, 0
total, 63707, 11011, 11011, 11011, 2.9, 2.6, 4.6,
11.2, 20.7, 62.6, 9.0, 0.14386, 0, 0, 0, 0,
0, 0
total, 74246, 10539, 10539, 10539, 3.0, 2.7, 5.2,
11.9, 41.4, 46.3, 10.0, 0.13055, 0, 0, 0, 0,
0, 0
total, 84693, 10447, 10447, 10447, 3.0, 2.8, 5.0,
7.2, 13.6, 30.0, 11.0, 0.11879, 0, 0, 0, 0,
0, 0
total, 95304, 10611, 10611, 10611, 3.0, 2.8, 5.0,
7.2, 14.6, 32.6, 12.0, 0.10912, 0, 0, 0, 0,
0, 0
total, 105865, 10561, 10561, 10561, 3.0, 2.7, 4.8,
10.4, 53.1, 60.6, 13.0, 0.10111, 0, 0, 0, 0,
0, 0
total, 116431, 10566, 10566, 10566, 3.0, 2.8, 5.0,
8.9, 14.8, 25.2, 14.0, 0.09380, 0, 0, 0, 0,
0, 0
total, 127856, 11425, 11425, 11425, 2.8, 2.6, 4.5,
6.8, 16.6, 24.4, 15.0, 0.08816, 0, 0, 0, 0,
0, 0
total, 139136, 11280, 11280, 11280, 2.8, 2.6, 4.4,
6.5, 17.6, 20.6, 16.0, 0.08292, 0, 0, 0, 0,
0, 0
total, 149943, 10807, 10807, 10807, 2.9, 2.7, 4.5,
6.1, 50.0, 52.2, 17.0, 0.07817, 0, 0, 0, 0,
0, 0
total, 161338, 11395, 11395, 11395, 2.8, 2.6, 4.8,
10.1, 18.4, 30.0, 18.0, 0.07411, 0, 0, 0, 0,
0, 0
total, 172127, 10789, 10789, 10789, 2.9, 2.6, 5.3,
12.3, 41.9, 53.9, 19.0, 0.07025, 0, 0, 0, 0,
0, 0
total, 182078, 9951, 9951, 9951, 3.2, 2.1, 10.9,
20.9, 36.3, 80.7, 20.0, 0.06662, 0, 0, 0, 0,
0, 0
total, 193140, 11062, 11062, 11062, 2.9, 2.5, 5.3,
12.0, 29.0, 44.7, 21.0, 0.06354, 0, 0, 0, 0,
0, 0
total, 203536, 10396, 10396, 10396, 3.1, 2.6, 5.2,
11.7, 60.6, 68.7, 22.0, 0.06061, 0, 0, 0, 0,
0, 0
total, 214575, 11039, 11039, 11039, 2.9, 2.6, 5.0,
10.1, 22.5, 36.2, 23.0, 0.05801, 0, 0, 0, 0,
0, 0
total, 225072, 10497, 10497, 10497, 3.0, 2.6, 5.6,
13.2, 29.2, 48.2, 24.0, 0.05553, 0, 0, 0, 0,
0, 0
total, 235155, 10083, 10083, 10083, 3.1, 2.7, 4.8,
9.9, 89.3, 102.6, 25.0, 0.05329, 0, 0, 0, 0,
0, 0
total, 245257, 10102, 10102, 10102, 3.1, 2.6, 5.2,
11.4, 65.6, 67.3, 26.0, 0.05118, 0, 0, 0, 0,
0, 0
total, 256802, 11545, 11545, 11545, 2.7, 2.6, 4.5,
8.2, 18.6, 36.6, 27.0, 0.04946, 0, 0, 0, 0,
0, 0
total, 267959, 11157, 11157, 11157, 2.8, 2.6, 4.6,
9.2, 20.9, 69.1, 28.0, 0.04784, 0, 0, 0, 0,
0, 0
total, 279085, 11126, 11126, 11126, 2.8, 2.6, 4.6,
8.2, 16.0, 30.8, 29.0, 0.04620, 0, 0, 0, 0,
0, 0
total, 289773, 10688, 10688, 10688, 3.0, 2.7, 4.9,
7.9, 19.0, 35.5, 30.0, 0.04463, 0, 0, 0, 0,
0, 0
total, 299929, 10156, 10156, 10156, 3.1, 2.7, 4.7,
10.8, 87.3, 90.8, 31.0, 0.04316, 0, 0, 0, 0,
0, 0
total, 310716, 10787, 10787, 10787, 2.9, 2.7, 5.0,
8.9, 16.3, 29.2, 32.0, 0.04179, 0, 0, 0, 0,
0, 0
total, 320931, 10215, 10215, 10215, 2.8, 2.6, 4.5,
7.2, 18.4, 25.5, 33.0, 0.04050, 0, 0, 0, 0,
0, 0
total, 322021, 1090, 1090, 1090, 31.9, 3.1, 11.0,
974.1, 984.6, 985.1, 34.0, 0.15458, 0, 0, 0, 0,
0, 0
total, 333309, 11288, 11288, 11288, 2.8, 2.6, 4.7,
9.9, 21.0, 25.4, 35.0, 0.15035, 0, 0, 0, 0,
0, 0
total, 344239, 10930, 10930, 10930, 2.9, 2.6, 5.0,
9.9, 39.6, 54.8, 36.0, 0.14636, 0, 0, 0, 0,
0, 0
total, 355560, 11321, 11321, 11321, 2.8, 2.6, 4.7,
9.3, 22.8, 41.4, 37.0, 0.14249, 0, 0, 0, 0,
0, 0
total, 364990, 9430, 9430, 9430, 3.4, 2.6, 5.0,
12.3, 141.3, 158.2, 38.0, 0.13902, 0, 0, 0, 0,
0, 0
total, 376297, 11307, 11307, 11307, 2.8, 2.6, 4.5,
8.6, 21.8, 24.0, 39.0, 0.13559, 0, 0, 0, 0,
0, 0
total, 387651, 11354, 11354, 11354, 2.8, 2.5, 4.7,
8.9, 20.3, 26.4, 40.0, 0.13230, 0, 0, 0, 0,
0, 0
total, 394128, 6477, 6477, 6477, 2.7, 2.6, 4.3,
5.8, 12.0, 14.0, 41.0, 0.13099, 0, 0, 0, 0,
0, 0
total, 396263, 2135, 2135, 2135, 21.6, 2.7, 4.7,
1256.2, 1258.3, 1259.3, 42.0, 0.13955, 0, 0, 0, 0,
0, 0
total, 407132, 10869, 10869, 10869, 2.9, 2.6, 4.7,
10.1, 18.8, 24.2, 43.0, 0.13638, 0, 0, 0, 0,
0, 0
total, 418276, 11144, 11144, 11144, 2.8, 2.6, 4.6,
9.0, 15.6, 26.1, 44.0, 0.13327, 0, 0, 0, 0,
0, 0
total, 429779, 11503, 11503, 11503, 2.8, 2.6, 4.3,
5.6, 13.4, 22.0, 45.0, 0.13022, 0, 0, 0, 0,
0, 0
total, 439888, 10109, 10109, 10109, 3.1, 2.6, 4.5,
9.7, 106.7, 108.4, 46.0, 0.12741, 0, 0, 0, 0,
0, 0
total, 450854, 10966, 10966, 10966, 2.9, 2.6, 4.7,
8.5, 21.0, 40.6, 47.0, 0.12470, 0, 0, 0, 0,
0, 0
total, 462404, 11550, 11550, 11550, 2.7, 2.6, 4.3,
5.9, 11.4, 19.3, 48.0, 0.12202, 0, 0, 0, 0,
0, 0
total, 472649, 10245, 10245, 10245, 2.7, 2.5, 4.3,
8.8, 13.8, 43.7, 49.0, 0.11969, 0, 0, 0, 0,
0, 0
total, 476297, 3648, 3648, 3648, 9.8, 2.7, 4.3,
12.1, 804.3, 805.3, 50.0, 0.11649, 0, 0, 0, 0,
0, 0
total, 487422, 11125, 11125, 11125, 2.8, 2.7, 4.5,
6.5, 16.0, 32.0, 51.0, 0.11421, 0, 0, 0, 0,
0, 0
total, 498707, 11285, 11285, 11285, 2.8, 2.6, 4.4,
8.9, 22.1, 32.6, 52.0, 0.11200, 0, 0, 0, 0,
0, 0
total, 509903, 11196, 11196, 11196, 2.8, 2.6, 4.5,
9.0, 16.5, 33.3, 53.0, 0.10988, 0, 0, 0, 0,
0, 0
