On Fri, Sep 6, 2019 at 5:56 PM Sergei Golubchik <[email protected]> wrote:
> Hi, Tom! > > On Sep 06, Tom Ryan wrote: > > > > Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 > thread_stack 0x49000 > > Sep 6 14:37:59 node-2 mysqld[1846]: > /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e] > > Sep 6 14:37:59 node-2 mysqld[1846]: > /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07] > > Sep 6 14:37:59 node-2 mysqld[1846]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890] > > Sep 6 14:37:59 node-2 mysqld[1846]: > /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd] > > Sep 6 14:37:59 node-2 mysqld[1846]: > /usr/sbin/mysqld(+0x566623)[0x5644f6973623] > > Sep 6 14:37:59 node-2 mysqld[1846]: > /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8] > > This looks truncated. Could you post a complete stack trace please? > Sep 6 14:37:59 node-2 mysqld[1846]: 190906 14:37:59 [ERROR] mysqld got signal 11 ; Sep 6 14:37:59 node-2 mysqld[1846]: This could be because you hit a bug. It is also possible that this binary Sep 6 14:37:59 node-2 mysqld[1846]: or one of the libraries it was linked against is corrupt, improperly built, Sep 6 14:37:59 node-2 mysqld[1846]: or misconfigured. This error can also be caused by malfunctioning hardware. Sep 6 14:37:59 node-2 mysqld[1846]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Sep 6 14:37:59 node-2 mysqld[1846]: We will try our best to scrape up some info that will hopefully help Sep 6 14:37:59 node-2 mysqld[1846]: diagnose the problem, but since we have already crashed, Sep 6 14:37:59 node-2 mysqld[1846]: something is definitely wrong and this may fail. Sep 6 14:37:59 node-2 mysqld[1846]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~xenial-log Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size=134217728 Sep 6 14:37:59 node-2 mysqld[1846]: read_buffer_size=2097152 Sep 6 14:37:59 node-2 mysqld[1846]: max_used_connections=30 Sep 6 14:37:59 node-2 mysqld[1846]: max_threads=4002 Sep 6 14:37:59 node-2 mysqld[1846]: thread_count=45 Sep 6 14:37:59 node-2 mysqld[1846]: It is possible that mysqld could use up to Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K bytes of memory Sep 6 14:37:59 node-2 mysqld[1846]: Hope that's ok; if not, decrease some variables in the equation. Sep 6 14:37:59 node-2 mysqld[1846]: Thread pointer: 0x7f32052a0348 Sep 6 14:37:59 node-2 mysqld[1846]: Attempting backtrace. You can use the following information to find out Sep 6 14:37:59 node-2 mysqld[1846]: where mysqld died. If you see no messages after this, something went Sep 6 14:37:59 node-2 mysqld[1846]: terribly wrong... Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 thread_stack 0x49000 Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1cfa)[0x5644f6edecfa] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1fa3)[0x5644f6edefa3] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x12f)[0x5644f6c75acf] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x22)[0x5644f6d988e2] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x164)[0x5644f6a97034] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xc9f)[0x5644f6ab85af] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5644f6ab8833] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x5644 f6ab6b26] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x5644f6ab7456] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x55756d)[0x5644f696456d] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7371)[0x5644f6a61741] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x5644f6a632da] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x656b59)[0x5644f6a63b59] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2633)[0x5644f6a66ae3] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5644f6a670ac] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x281)[0x5644f6b43691] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5644f6b437bd] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f33e14126db] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f33e081088f] Sep 6 14:37:59 node-2 mysqld[1846]: Trying to get some variables. Sep 6 14:37:59 node-2 mysqld[1846]: Some pointers may be invalid and cause the dump to abort. Sep 6 14:37:59 node-2 mysqld[1846]: Query (0x7f3205209000): SELECT field_data_field_weight.field_weight_value AS field_data_field_weight_field_weight_value, node.nid AS nid, DATE_FORMAT((DATE_A DD('19700101', INTERVAL node.created SECOND) + INTERVAL -14400 SECOND), '%Y%m') AS node_created_month FROM node node LEFT JOIN field_data_field_weight field_data_field_weight ON node.nid = fiel d_data_field_weight.entity_id AND (field_data_field_weight.entity_type = 'node' AND field_data_field_weight.deleted = '0') WHERE (( (node.status = 1 OR (node.uid = 0 AND 0 <> 0 AND 0 = 1) OR 0 = 1) AND (node.nid IN (SELECT tn.nid AS nid FROM taxonomy_index tn WHERE ( (tn.tid = '3') ))) ))AND ( EXISTS (SELECT na.nid AS nid FROM node_access na WHERE (( (na.gid = '0') AND (na.realm = 'all') )OR( (na.gid = '0') AND (na.realm = 'content_access_author') )OR( (na.gid = '1') AND (na.realm = 'content_access_rid') ))AND (na.grant_view >= '1') AND (node.nid = na.nid) )) ORDER BY no de_created_month DESC, field_data_field_weight_field_weight_value DESC LIMIT 15 OFFSET 0 Sep 6 14:37:59 node-2 mysqld[1846]: Connection ID (thread ID): 15972 Sep 6 14:37:59 node-2 mysqld[1846]: Status: NOT_KILLED Sep 6 14:37:59 node-2 mysqld[1846]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_conditio n_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_ma tch_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on, optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subqu ery=on,rowid_filter=on,condition_pushdown_from_having=on Sep 6 14:37:59 node-2 mysqld[1846]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Sep 6 14:37:59 node-2 mysqld[1846]: information that should help you find out what is causing the crash. Sep 6 14:37:59 node-2 mysqld[1846]: Writing a core file... Sep 6 14:37:59 node-2 mysqld[1846]: Working directory at /var/lib/mysql Sep 6 14:37:59 node-2 mysqld[1846]: Resource Limits: Sep 6 14:37:59 node-2 mysqld[1846]: Limit Soft Limit Hard Limit Units Sep 6 14:37:59 node-2 mysqld[1846]: Max cpu time unlimited unlimited seconds Sep 6 14:37:59 node-2 mysqld[1846]: Max file size unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max data size unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max stack size 8388608 unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max core file size 0 unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max resident set unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max processes 128431 128431 processes Sep 6 14:37:59 node-2 mysqld[1846]: Max open files 16364 16364 files Sep 6 14:37:59 node-2 mysqld[1846]: Max locked memory 16777216 16777216 bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max address space unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max file locks unlimited unlimited locks Sep 6 14:37:59 node-2 mysqld[1846]: Max pending signals 128431 128431 signals Sep 6 14:37:59 node-2 mysqld[1846]: Max msgqueue size 819200 819200 bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max nice priority 0 0 Sep 6 14:37:59 node-2 mysqld[1846]: Max realtime priority 0 0 Sep 6 14:37:59 node-2 mysqld[1846]: Max realtime timeout unlimited unlimited us Sep 6 14:37:59 node-2 mysqld[1846]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P Sep 6 14:37:59 node-2 kernel: [31368.222207] show_signal_msg: 7 callbacks suppressed Sep 6 14:37:59 node-2 kernel: [31368.222210] mysqld[3138]: segfault at 0 ip 00005644f6c7b1fd sp 00007f32714417b0 error 4 here's one from this morning Sep 7 05:12:07 node-1 mysqld[18950]: This could be because you hit a bug. It is also possible that this binary Sep 7 05:12:07 node-1 mysqld[18950]: or one of the libraries it was linked against is corrupt, improperly built, Sep 7 05:12:07 node-1 mysqld[18950]: or misconfigured. This error can also be caused by malfunctioning hardware. Sep 7 05:12:07 node-1 mysqld[18950]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Sep 7 05:12:07 node-1 mysqld[18950]: We will try our best to scrape up some info that will hopefully help Sep 7 05:12:07 node-1 mysqld[18950]: diagnose the problem, but since we have already crashed, Sep 7 05:12:07 node-1 mysqld[18950]: something is definitely wrong and this may fail. Sep 7 05:12:07 node-1 mysqld[18950]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~bionic-log Sep 7 05:12:07 node-1 mysqld[18950]: key_buffer_size=134217728 Sep 7 05:12:07 node-1 mysqld[18950]: read_buffer_size=2097152 Sep 7 05:12:07 node-1 mysqld[18950]: max_used_connections=27 Sep 7 05:12:07 node-1 mysqld[18950]: max_threads=4002 Sep 7 05:12:07 node-1 mysqld[18950]: thread_count=42 Sep 7 05:12:07 node-1 mysqld[18950]: It is possible that mysqld could use up to Sep 7 05:12:07 node-1 mysqld[18950]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K bytes of memory Sep 7 05:12:07 node-1 mysqld[18950]: Hope that's ok; if not, decrease some variables in the equation. Sep 7 05:12:07 node-1 mysqld[18950]: Thread pointer: 0x7f1adc00b618 Sep 7 05:12:07 node-1 mysqld[18950]: Attempting backtrace. You can use the following information to find out Sep 7 05:12:07 node-1 mysqld[18950]: where mysqld died. If you see no messages after this, something went Sep 7 05:12:07 node-1 mysqld[18950]: terribly wrong... Sep 7 05:12:07 node-1 mysqld[18950]: stack_bottom = 0x7f199fecadd8 thread_stack 0x49000 Sep 7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a2e3675cee] Sep 7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55a2e30ed085] Sep 7 05:12:07 node-1 mysqld[18950]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f1b115d0890] Sep 7 05:12:07 node-1 kernel: [84233.145921] mysqld[26579]: segfault at 0 ip 000055a2e30f7500 sp 00007f19905b9e10 error 4 in mysqld[55a2e2896000+13d6000] Sep 7 05:12:07 node-1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV Sep 7 05:12:07 node-1 systemd[1]: mariadb.service: Failed with result 'signal'. > > And please tell exactly what binaries you're using. Version, > architecture, Linux distribution. > > ubuntu 18.04 mariadb respository /usr/sbin/mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=cdf04892e30322f0ee59a1aa604e6087ad040231, stripped > > This is getting old and is unlike our usual mariadb experience. > > > > Looking at the jira for 10.4.8, it appears to fix many problems in > > 10.4 and I'm hopeful it solves this. > > > > In the meantime, any ideas or suggestions for things to look for? > > 10.4.8 should be out any day now. But if I'd know more about the crash, > I could tell whether it's fixed there or not. > > Regards, > Sergei >
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
