Yeah had you not discovered it being a mysql problem, I would have next
suggested you verify system memory / cpu with something like memtest.
Glad to hear you resolved the problem.
-cf
On Tue, Nov 29, 2016 at 7:33 AM, Robbins, Lewis <lewis.robb...@umassmed.edu>
wrote:
> This was a mysql problem. A little more heavy-handed than I would have
> preferred, but a complete re-install of mysql resolved the issue.
>
>
> Thanks,
>
> Lewis
>
>
> Lewis Robbins | Systems Admin Engineer Sr.
> Data Sciences & Technology
> lewis.robb...@umassmed.edu
> O. 774-455-4353
>
>
> Confidentiality Notice:
> This e-mail message, including any attachments, is for the sole use of the
> intended recipient(s) and may contain confidential, proprietary and
> privileged information. Any unauthorized review, use, disclosure or
> distribution is prohibited. If you are not the intended recipient, please
> contact the sender immediately and destroy or permanently delete all copies
> of the original message.
>
>
> ------------------------------
> *From:* Robbins, Lewis <lewis.robb...@umassmed.edu>
> *Sent:* Monday, November 28, 2016 4:21 PM
> *To:* Colin Faber
>
> *Cc:* robinhood-supp...@lists.sf.net
> *Subject:* Re: [robinhood-support] scan causing mysql to crash
>
>
> I'm leaning toward it being a MySQL issue as well, just performed an
> identical robinhood install on a different system and it seems to be
> working correctly. For what it's worth, I had tried a MySQL update, but it
> didn't seem to help.
>
>
> Any MySQL guru's have thoughts on where to look?
>
>
> Thanks,
>
> Lewis
>
>
> Lewis Robbins | Systems Admin Engineer Sr.
> Data Sciences & Technology
> lewis.robb...@umassmed.edu
> O. 774-455-4353
>
>
> Confidentiality Notice:
> This e-mail message, including any attachments, is for the sole use of the
> intended recipient(s) and may contain confidential, proprietary and
> privileged information. Any unauthorized review, use, disclosure or
> distribution is prohibited. If you are not the intended recipient, please
> contact the sender immediately and destroy or permanently delete all copies
> of the original message.
> ------------------------------
> *From:* Colin Faber <cfa...@gmail.com>
> *Sent:* Monday, November 28, 2016 4:17:00 PM
> *To:* Robbins, Lewis
> *Cc:* robinhood-supp...@lists.sf.net
> *Subject:* Re: [robinhood-support] scan causing mysql to crash
>
>
> This seems more like a MySQL issue, have you checked to see if there is an
> update for MySQL?
>
> On Nov 28, 2016 1:46 PM, "Robbins, Lewis" <lewis.robb...@umassmed.edu>
> wrote:
>
>> Hello,
>>
>> We had been successfully running robinhood-tmps 2.5.x for about a year,
>> and recently started having robinhood/mysql issues, where a scan will crash
>> the mysql db.
>>
>>
>> I have tried dropping and recreating the robinhood database, as well a
>> fresh install to robinhood 3.0 but am still having the same issue. It's a
>> pretty generic scan of a posix file system (NFS if it matters).
>>
>>
>> Any tips would be greatly appreciated.
>>
>>
>> The mysql log contains messages about performing a backtrace, and seems
>> to be having a problem with an invalid pointer.
>>
>>
>> Thread pointer: 0xfa9165c0
>> Attempting backtrace. You can use the following information to find out
>> where mysqld died. If you see no messages after this, something went
>> terribly wrong...
>> stack_bottom = 7ff104410d98 thread_stack 0x40000
>> /usr/libexec/mysqld(my_print_stacktrace+0x29) [0x8509b9]
>> /usr/libexec/mysqld(handle_fatal_signal+0x483) [0x6a3ef3]
>> /lib64/libpthread.so.0() [0x30ed20f7e0]
>> /lib64/libc.so.6(gsignal+0x35) [0x30ec632625]
>> /lib64/libc.so.6(abort+0x175) [0x30ec633e05]
>> /usr/libexec/mysqld() [0x76c751]
>> /usr/libexec/mysqld() [0x76d630]
>> /usr/libexec/mysqld(fseg_create_general+0x390) [0x76dc10]
>> /usr/libexec/mysqld(trx_undo_assign_undo+0x6ac) [0x7df85c]
>> /usr/libexec/mysqld(trx_undo_report_row_operation+0x1b5f) [0x7d31af]
>> /usr/libexec/mysqld(btr_cur_update_in_place+0x1ae) [0x73c3fe]
>> /usr/libexec/mysqld(btr_cur_optimistic_update+0x34f) [0x73d00f]
>> /usr/libexec/mysqld() [0x7bce52]
>> /usr/libexec/mysqld() [0x7bfbf1]
>> /usr/libexec/mysqld(row_upd_step+0x317) [0x7c0967]
>> /usr/libexec/mysqld(row_update_for_mysql+0x1b9) [0x7ac719]
>> /usr/libexec/mysqld(ha_innobase::update_row(unsigned char const*,
>> unsigned char*)+0x4a0) [0x731980]
>> /usr/libexec/mysqld(handler::ha_update_row(unsigned char const*,
>> unsigned char*)+0x6e) [0x699c5e]
>> /usr/libexec/mysqld(mysql_update(THD*, TABLE_LIST*, List<Item>&,
>> List<Item>&, Item*, unsigned int, st_order*, unsigned long long,
>> enum_duplicates, bool)+0x114b) [0x64971b]
>> /usr/libexec/mysqld(mysql_execute_command(THD*)+0x963) [0x5ca4b3]
>> /usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, char
>> const**)+0x2d3) [0x5cf5b3]
>> /usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*,
>> unsigned int)+0xd4b) [0x5d168b]
>> /usr/libexec/mysqld(do_command(THD*)+0xea) [0x5d220a]
>> /usr/libexec/mysqld(handle_one_connection+0x23e) [0x5c548e]
>> /lib64/libpthread.so.0() [0x30ed207aa1]
>> /lib64/libc.so.6(clone+0x6d) [0x30ec6e893d]
>>
>> Trying to get some variables.
>> Some pointers may be invalid and cause the dump to abort.
>> Query (7fcda0004a00): is an invalid pointer
>> .
>> .
>> .
>> .
>>
>>
>>
>> The snippet of the robinhood log and configuration files are below:
>>
>>
>> # robinhood --scan --once -L stderr
>> Using config file '/etc/robinhood.d/s4s.conf'.
>> 2016/11/28 14:43:26 [10439/1] CheckFS | '/s4s' matches mount point
>> '/s4s', type=nfs, fs=ghpccavere.umassrc.org:/
>> 2016/11/28 14:43:26 [10439/1] ListMgr | Table does not exist: 'SELECT
>> value FROM VARS WHERE varname='VersionFunctionSet'' (Table
>> 'robinhood_s4s.VARS' doesn't exist)
>> 2016/11/28 14:43:26 [10439/1] ListMgr | No function versioning (expected:
>> 1.5). Existing functions will be dropped and re-created.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | Table does not exist: 'SELECT
>> value FROM VARS WHERE varname='VersionTriggerSet'' (Table
>> 'robinhood_s4s.VARS' doesn't exist)
>> 2016/11/28 14:43:26 [10439/1] ListMgr | No trigger versioning (expected:
>> 1.4). Existing triggers will be dropped and re-created.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | table VARS does not exist (or
>> wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | table ENTRIES does not exist (or
>> wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | table NAMES does not exist (or
>> wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | table ANNEX_INFO does not exist
>> (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | function sz_range does not exist
>> (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | table ACCT_STAT does not exist
>> (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | Populating accounting table from
>> existing DB contents. This can take a while...
>> 2016/11/28 14:43:26 [10439/1] ListMgr | table SOFT_RM does not exist (or
>> wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | trigger ACCT_ENTRY_INSERT does
>> not exist (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | trigger ACCT_ENTRY_DELETE does
>> not exist (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | trigger ACCT_ENTRY_UPDATE does
>> not exist (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | function one_path does not exist
>> (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] ListMgr | function this_path does not exist
>> (or wrong version): creating it.
>> 2016/11/28 14:43:26 [10439/1] EntryProc | No fileclass defined in
>> configuration, disabling fileclass matching.
>> 2016/11/28 14:43:26 [10439/2] FS_Scan | Starting scan of /s4s
>> 2016/11/28 14:43:26 [10439/2] FS_Scan | Notice: this is the first scan
>> (DB is empty)
>> 2016/11/28 14:43:27 [10439/4] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/3] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/8] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/7] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/4] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/7] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/9] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/10] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/9] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/6] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/8] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/11] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/11] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/5] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/10] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/6] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/8] ListMgr | DB connection error 2013
>> 2016/11/28 14:43:27 [10439/3] ListMgr | Retryable DB error in
>> ListMgr_BatchInsert l.334. Retrying...
>> 2016/11/28 14:43:27 [10439/5] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:27 [10439/8] ListMgr | Retryable DB error in
>> ListMgr_Exists l.49. Retrying...
>> 2016/11/28 14:43:28 [10439/11] ListMgr | DB connection error 2002
>> 2016/11/28 14:43:28 [10439/4] ListMgr | DB connection error 2002
>> 2016/11/28 14:43:28 [10439/3] ListMgr | DB connection error 2006
>> 2016/11/28 14:43:28 [10439/9] ListMgr | DB connection error 2002
>>
>> $ cat /etc/robinhood.d/s4s.conf
>> %include "includes/common.conf"
>>
>> General {
>> fs_path = "/s4s";
>> fs_type = nfs;
>> stay_in_fs = FALSE;
>> }
>> ListManager {
>> MySQL {
>> server = localhost;
>> db = robinhood_s4s;
>> user = robinhood;
>> password_file = /etc/robinhood.d/.dbpassword;
>> }
>> }
>>
>> $cat /etc/robinhood.d/includes/common.conf
>> Log {
>> log_file = "/var/log/robinhood/tmp_fs.log";
>> report_file = "/var/log/robinhood/reports.log";
>> alert_file = "/var/log/robinhood/alerts.log";
>>
>> }
>> FS_Scan {
>> Ignore {
>> #don't scan ".snapshot" directories
>> type == directory
>> and
>> name == ".snapshot"
>> }
>>
>> #completion_command = "/root/Scripts/s4s_useage.pl -f
>> /etc/robinhood.d/tmpfs/s4s.conf -p /s4s";
>> #scan_interval = 24h ;
>> }
>> purge_policies {
>> Ignore { owner == * }
>> }
>>
>>
>> Thanks,
>>
>> Lewis
>>
>>
>> Lewis Robbins | Systems Admin Engineer Sr.
>> Data Sciences & Technology
>> lewis.robb...@umassmed.edu
>> O. 774-455-4353
>>
>>
>> Confidentiality Notice:
>> This e-mail message, including any attachments, is for the sole use of
>> the intended recipient(s) and may contain confidential, proprietary and
>> privileged information. Any unauthorized review, use, disclosure or
>> distribution is prohibited. If you are not the intended recipient, please
>> contact the sender immediately and destroy or permanently delete all copies
>> of the original message.
>>
>> ------------------------------------------------------------
>> ------------------
>>
>> _______________________________________________
>> robinhood-support mailing list
>> robinhood-support@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/robinhood-support
>>
>>
> ------------------------------------------------------------
> ------------------
>
> _______________________________________________
> robinhood-support mailing list
> robinhood-support@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/robinhood-support
>
>
------------------------------------------------------------------------------
_______________________________________________
robinhood-support mailing list
robinhood-support@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/robinhood-support
