Public bug reported:
Hello, a few times a day there is such a problem on slave database which
sometimes stops working database. Could this be an error in the
application? centos-release-7-3.1611.el7.centos.x86_64, Server version:
5.6.30-76.3-56-log Percona XtraDB Cluster (GPL), Release rel76.3,
Revision 64742ac, WSREP version 25.16, wsrep_25.16
=====================================
2018-02-13 10:30:22 7fb25e350700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 20 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 521370 srv_active, 0 srv_shutdown, 49974 srv_idle
srv_master_thread log flush and writes: 571344
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 3873624
--Thread 140403991103232 has waited at fts0fts.cc line 3540 for 948.00 seconds
the semaphore:
X-lock on RW-latch at 0x3214368 '&cache->lock'
a writer (thread id 140404002690816) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file not yet reserved line 0
Last time write locked in file
/mnt/workspace/percona-xtradb-cluster-5.6-redhat-binary-new/label_exp/centos7-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.6.30-76.3/storage/innobase/fts/fts0fts.cc
line 4539
OS WAIT ARRAY INFO: signal count 4313594
Mutex spin waits 5446767, rounds 31829194, OS waits 627714
RW-shared spins 4490432, rounds 106775325, OS waits 3072745
RW-excl spins 707182, rounds 8258334, OS waits 135026
Spin rounds per wait: 5.84 mutex, 23.78 RW-shared, 11.68 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 116465414850
Purge done for trx's n:o < 116465414850 undo n:o < 0 state: running but idle
History list length 2314
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 116465414848, not started
MySQL thread id 77354, OS thread handle 0x7fb25a140700, query id 0 Waiting for
master to send event
---TRANSACTION 116465414815, ACTIVE (PREPARED) 948 sec committing
3 lock struct(s), heap size 360, 1 row lock(s), undo log entries 2
MySQL thread id 77355, OS thread handle 0x7fb25a03c700, query id 61610521
System lock
---TRANSACTION 116465414814, ACTIVE 948 sec doing SYNC index
5 lock struct(s), heap size 1184, 0 row lock(s), undo log entries 36742
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (read thread)
I/O thread 7 state: waiting for completed aio requests (read thread)
I/O thread 8 state: waiting for completed aio requests (read thread)
I/O thread 9 state: waiting for completed aio requests (read thread)
I/O thread 10 state: waiting for completed aio requests (write thread)
I/O thread 11 state: waiting for completed aio requests (write thread)
I/O thread 12 state: waiting for completed aio requests (write thread)
I/O thread 13 state: waiting for completed aio requests (write thread)
I/O thread 14 state: waiting for completed aio requests (write thread)
I/O thread 15 state: waiting for completed aio requests (write thread)
I/O thread 16 state: waiting for completed aio requests (write thread)
I/O thread 17 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0,
0, 0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
163572273 OS file reads, 41431538 OS file writes, 30217350 OS fsyncs
1 pending preads, 0 pending pwrites
15.35 reads/s, 16384 avg bytes/read, 22.65 writes/s, 5.45 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 33942, seg size 33944, 753454 merges
merged operations:
insert 1898017, delete mark 9489470, delete 2760647
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 17.65 non-hash searches/s
---
LOG
---
Log sequence number 35144499382571
Log flushed up to 35144499370804
Pages flushed up to 35144499361152
Last checkpoint at 35144499361152
Max checkpoint age 3477268685
Checkpoint age target 3368604039
Modified age 21419
Checkpoint age 21419
0 pending log writes, 0 pending chkp writes
27139038 log i/o's done, 2.60 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 107307073536; in additional pool allocated 0
Total memory allocated by read views 344
Internal hash tables (constant factor + variable factor)
Adaptive hash index 1631983792 (1631979592 + 4200)
Page hash 8851208 (buffer pool 0 only)
Dictionary cache 414926219 (407996432 + 6929787)
File system 1521232 (812272 + 708960)
Lock system 254998696 (254997976 + 720)
Recovery system 0 (0 + 0)
Dictionary memory allocated 6929787
Buffer pool size 6291444
Buffer pool size, bytes 103079018496
Free buffers 12282
Database pages 6279162
Old database pages 2317650
Modified db pages 38
Pending reads 1
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 12532817, not young 1147053122
1.60 youngs/s, 16.50 non-youngs/s
Pages read 163570487, created 2167490, written 13386060
15.35 reads/s, 0.30 creates/s, 19.10 writes/s
Buffer pool hit rate 857 / 1000, young-making rate 15 / 1000 not 154 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 6279162, unzip_LRU len: 0
I/O sum[1019064]:cur[10368], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 6
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1029631, not young 96459518
0.05 youngs/s, 1.10 non-youngs/s
Pages read 13697458, created 188631, written 1241350
0.95 reads/s, 0.00 creates/s, 1.45 writes/s
Buffer pool hit rate 969 / 1000, young-making rate 1 / 1000 not 36 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1023
Database pages 523264
Old database pages 193138
Modified db pages 4
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1054973, not young 95930297
0.20 youngs/s, 1.20 non-youngs/s
Pages read 13627490, created 178481, written 1079858
1.15 reads/s, 0.00 creates/s, 1.40 writes/s
Buffer pool hit rate 766 / 1000, young-making rate 40 / 1000 not 244 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523264, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 2
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1031850, not young 94324134
0.20 youngs/s, 1.50 non-youngs/s
Pages read 13645467, created 176738, written 1063208
1.40 reads/s, 0.05 creates/s, 1.55 writes/s
Buffer pool hit rate 177 / 1000, young-making rate 117 / 1000 not 882 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 4
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1049058, not young 97302440
0.20 youngs/s, 1.95 non-youngs/s
Pages read 13549649, created 178001, written 1097802
1.70 reads/s, 0.00 creates/s, 1.70 writes/s
Buffer pool hit rate 228 / 1000, young-making rate 90 / 1000 not 886 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1022
Database pages 523265
Old database pages 193139
Modified db pages 3
Pending reads 1
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1015775, not young 95774195
0.15 youngs/s, 1.75 non-youngs/s
Pages read 13740361, created 191329, written 1227618
1.70 reads/s, 0.00 creates/s, 1.90 writes/s
Buffer pool hit rate 774 / 1000, young-making rate 20 / 1000 not 233 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523265, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 1
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1064826, not young 93792472
0.05 youngs/s, 1.55 non-youngs/s
Pages read 13559905, created 178166, written 1090371
1.50 reads/s, 0.00 creates/s, 1.65 writes/s
Buffer pool hit rate 91 / 1000, young-making rate 30 / 1000 not 939 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 3
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1036260, not young 95612049
0.05 youngs/s, 1.15 non-youngs/s
Pages read 13688644, created 178582, written 1072266
1.10 reads/s, 0.05 creates/s, 2.15 writes/s
Buffer pool hit rate 948 / 1000, young-making rate 2 / 1000 not 54 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 3
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1067419, not young 96344906
0.15 youngs/s, 1.40 non-youngs/s
Pages read 13535153, created 178279, written 1115582
1.25 reads/s, 0.00 creates/s, 1.45 writes/s
Buffer pool hit rate 680 / 1000, young-making rate 38 / 1000 not 358 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 8
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1022
Database pages 523265
Old database pages 193139
Modified db pages 7
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1027384, not young 96490251
0.25 youngs/s, 1.35 non-youngs/s
Pages read 13726078, created 187815, written 1228430
1.15 reads/s, 0.20 creates/s, 1.90 writes/s
Buffer pool hit rate 824 / 1000, young-making rate 38 / 1000 not 207 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523265, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 9
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 1
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1070867, not young 95583412
0.20 youngs/s, 0.75 non-youngs/s
Pages read 13595580, created 176502, written 1057298
0.65 reads/s, 0.00 creates/s, 1.05 writes/s
Buffer pool hit rate 971 / 1000, young-making rate 9 / 1000 not 34 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 10
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1023
Database pages 523264
Old database pages 193138
Modified db pages 3
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1035655, not young 93920310
0.10 youngs/s, 1.90 non-youngs/s
Pages read 13666207, created 176585, written 1039196
1.90 reads/s, 0.00 creates/s, 2.00 writes/s
Buffer pool hit rate 96 / 1000, young-making rate 47 / 1000 not 904 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523264, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
---BUFFER POOL 11
Buffer pool size 524287
Buffer pool size, bytes 8589918208
Free buffers 1024
Database pages 523263
Old database pages 193137
Modified db pages 1
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1049119, not young 95519138
0.00 youngs/s, 0.90 non-youngs/s
Pages read 13538495, created 178381, written 1073081
0.90 reads/s, 0.00 creates/s, 0.90 writes/s
Buffer pool hit rate 685 / 1000, young-making rate 0 / 1000 not 315 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 523263, unzip_LRU len: 0
I/O sum[84922]:cur[864], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
2 RW transactions active inside InnoDB
0 RO transactions active inside InnoDB
2 out of 1000 descriptors used
Main thread process no. 26831, id 140404053047040, state: sleeping
Number of rows inserted 35202024, updated 300287776, deleted 31502321, read
111555713107
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
2018-02-13 10:30:28 7fb25eb51700 InnoDB: Assertion failure in thread
140404069832448 in file srv0srv.cc line 2181
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:30:28 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=142
max_threads=302
thread_count=5
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 251378 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x90bbdb]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x67c2e1]
/usr/lib64/libpthread.so.0(+0xf370)[0x7fcc0cd51370]
/usr/lib64/libc.so.6(gsignal+0x37)[0x7fcc0aed61d7]
/usr/lib64/libc.so.6(abort+0x148)[0x7fcc0aed78c8]
/usr/sbin/mysqld[0x9f5d01]
/usr/lib64/libpthread.so.0(+0x7dc5)[0x7fcc0cd49dc5]
/usr/lib64/libc.so.6(clone+0x6d)[0x7fcc0af9876d]
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find
information
in the manual which will help you identify the cause of the crash.
** Affects: percona-xtradb-cluster-5.6 (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1749190
Title:
InnoDB: Assertion failure in thread in file srv0srv.cc line 2181
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1749190/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
