Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Frustratingly, I still can't reproduce the error locally on a handful of long-lived RT instances all of which have been upgraded through 4.0.12. Hi Thomas, Thank you for spending the time looking into this. Seeing as how my last step of laying down a new install base corrected the matter on our RTPROD server I'm going to shelve this and call it good. I wish I could've narrowed it down to a specific cause and resolution however due to the number of higher priority projects screaming for attention I won't be able to put much more time into investigating this further. If for some reason you need/require additional information out of what I've provided so far, please let me know and I'd be happy to oblige when possible. Cheers, ~Pete_Jibe Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains. -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Hi Thomas,
Looks like the mail filter stopped this one from going through due to the
attached log file. Re-forwarding minus log file.
Also of note, yesterday evening I brought our RTPROD server offline for a
reboot. When the OS was back online I backed up the existing RTPROD
installation directory and re-ran the configuration to re-create the install
directory. I then copied over our RT_SiteConfig file and custom log and
started the web service back up. I re-tested the Global At A Glance setting
and found that the issue was no longer appearing.
Sonothing done on the database end (other than re-instating those user
records). For this particular problem it seems that me laying down a new
install base somehow corrected the original issue.
~Pete_Jibe
-Original Message-
From: Pete Beebe
Sent: Thursday, May 16, 2013 10:55 AM
To: 'rt-users@lists.bestpractical.com'
Subject: RE: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance
error
Hi Thomas,
Have you run rt-validator?
Log file attached. All references are likely due to the deletion of old user
records directly from mysql which we now know is not recommended.
What are the output of the 3 SQL queries I sent you earlier on this newly
upgraded dev instance?
All from the upgrade Oracle RT DEV instance:
select id, Name, Description, ObjectType, ObjectId from Attributes where
ObjectType='RT::System';
137 Search - My Tickets [_1] highest priority tickets I own
RT::System 1
138 Search - Unowned Tickets[_1] newest unowned tickets
RT::System 1
139 HomepageSettingsHomepageSettingsRT::System 1
140 Search - My Tickets [_1] highest priority tickets I own
RT::System 1
141 Search - Unowned Tickets[_1] newest unowned tickets
RT::System 1
142 HomepageSettingsHomepageSettingsRT::System 1
153 Search - My Tickets [_1] highest priority tickets I own
RT::System 1
154 Search - Unowned Tickets[_1] newest unowned tickets
RT::System 1
155 HomepageSettingsHomepageSettingsRT::System 1
300 Search - Bookmarked Tickets Bookmarked Tickets RT::System
1
318 BrandedSubjectTag Queue id = subject tag map RT::System
1
3021SavedSearch 10 highest priority tickets I own RT::System
1
3842QueueCacheNeedsUpdate 0 RT::System 1
select id, Name, Created, LastUpdated from Users where Name in ('RT_System',
'Nobody', 'root');
10 Nobody 06-SEP-05 06-SEP-05
1 RT_System 06-SEP-05 06-SEP-05
12 root06-SEP-05 15-MAY-13
select id, Domain, Type, Created, LastUpdated from Groups where Type in
('Everyone', 'Privileged', 'Unprivileged');
3 SystemInternal Everyone
4 SystemInternal Privileged
5 SystemInternal Unprivileged
As a note, the duplicate Search - Unowned Tickets duplicates are likely a
remnant of trying to fix a link issue in the previous 3.8.5 instance. Relevant
thread here: http://www.gossamer-threads.com/lists/rt/users/58712 It's a side
issue that was never satisfactorily resolved and would be preferable to clean
out the duplicates.
Here's the same SQL queries output from the original MySQL 3.8.5 database:
mysql select id, Name, Description, ObjectType, ObjectId from Attributes where
ObjectType='RT::System';
+--+-+-++--+
| id | Name| Description |
ObjectType | ObjectId |
+--+-+-++--+
| 137 | Search - My Tickets | [_1] highest priority tickets I own |
RT::System |1 |
| 138 | Search - Unowned Tickets| [_1] newest unowned tickets |
RT::System |1 |
| 139 | HomepageSettings| HomepageSettings|
RT::System |1 |
| 140 | Search - My Tickets | [_1] highest priority tickets I own |
RT::System |1 |
| 141 | Search - Unowned Tickets| [_1] newest unowned tickets |
RT::System |1 |
| 142 | HomepageSettings| HomepageSettings|
RT::System |1 |
| 153 | Search - My Tickets | [_1] highest priority tickets I own |
RT::System |1 |
| 154 | Search - Unowned Tickets| [_1] newest unowned tickets |
RT::System |1 |
| 155 | HomepageSettings| HomepageSettings|
RT::System |1 |
| 300 | Search - Bookmarked Tickets | Bookmarked Tickets |
RT::System |1 |
| 318 | BrandedSubjectTag | Queue id = subject tag map |
RT::System |1 |
| 3021 | SavedSearch | 10 highest priority tickets I own
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
On 05/17/2013 09:13 AM, Pete Beebe wrote: Hi Thomas, Looks like the mail filter stopped this one from going through due to the attached log file. Re-forwarding minus log file. It was briefly stopped, yes, but I let it through manually. There's a size limit on posts at which point they require manual moderation. Also of note, yesterday evening I brought our RTPROD server offline for a reboot. When the OS was back online I backed up the existing RTPROD installation directory and re-ran the configuration to re-create the install directory. I then copied over our RT_SiteConfig file and custom log and started the web service back up. I re-tested the Global At A Glance setting and found that the issue was no longer appearing. Sonothing done on the database end (other than re-instating those user records). For this particular problem it seems that me laying down a new install base somehow corrected the original issue. Out of curiosity, how did you reinstate the user records? The error was still occurring after you'd reinstated them, but before you reinstalled, right? I'm surprised that a new install would correct the issue, but it is possible. Basically a component was returning an unexpected empty value. Can you run `diff -ruN` against the two install trees (the one you backed up and the one currently in use)? Alternatively, sent two tarballs over and I can do it. Thanks for continuing to dig at this and all the info you've provided. Thomas -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Hi Thomas, Out of curiosity, how did you reinstate the user records? On the original RT 3.8.5 server I had a couple of mysql dumps available with the previously deleted records still intact. On the original server I copied one of the mysql backups to the RTDEV server, imported it into an empty MySQL rt database and then ran the upgrade to 4.0.12 followed by the migration to the Oracle schema. I then extracted the specific deleted records from the RTDEV Oracle schema and inserted them into the RTPROD User table followed by re-generating the User table sequence. If I was more savvy I probably could have just reconstructed the insert queries manually but the above method worked out well enough, just took a bit more time. The error was still occurring after you'd reinstated them, but before you reinstalled, right? This is correct. I re-instated the deleted user records into RTPROD and the Global RT At A Glance error was still occurring. It was only when I rebooted the RTPROD server and re-ran the base directory installation that the error cleared. What I didn't do which I should have done was re-test after the RTPROD reboot and before running the new base directory installation. Can you run `diff -ruN` against the two install trees (the one you backed up and the one currently in use)? Zip file is 190KB in size. I will be sending it separate from this reply. I cut out the session_data diff output to trim down the original log file (4MB). If you prefer that this additional diff info be included I can simply send you the zipped 4MB log file. ~Pete_Jibe Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains. -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
On 05/17/2013 01:36 PM, Pete Beebe wrote: Can you run `diff -ruN` against the two install trees (the one you backed up and the one currently in use)? Zip file is 190KB in size. I will be sending it separate from this reply. I cut out the session_data diff output to trim down the original log file (4MB). If you prefer that this additional diff info be included I can simply send you the zipped 4MB log file. The session data isn't necessary (and shouldn't be published publicly either; sessions contain private data). I filtered out irrelevant files (the new RT-Authen-ExternalAuth install, Mason cache files, documentation files (man and pod)) from the diff you sent, and it left no significant changes. Frustratingly, I still can't reproduce the error locally on a handful of long-lived RT instances all of which have been upgraded through 4.0.12. Thomas -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
[rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Hi Thomas, This might narrow down the issueI just noticed another thread titled RT at a Glance error after upgrade from 4.011 to 4.0.12 that also states the issue at hand. Just to be sure, I repeated the import on our RTDEV server using v4.0.11 of RT (using MySQL only, not Oracle) and confirmed that the Global RT At A Glance error does ~not~ occur. I will proceed with the migration to Oracle using 4.0.11 only and update this thread with the results. ~Pete_Jibe Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains. -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Hi Thomas, Just performed the RT migration from MySQL to Oracle using just 4.0.11 and re-tested the Global RT At A Glance error without issue. I will now repeat the 4.0.11 to 4.0.12 upgrade using just Oracle and update with results. ~Pete_Jibe Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains. -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
[rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Hi Thomas, The 4.0.11 to 4.0.12 Oracle upgrade completed and this RT At A Glance issue still did not reappear. This was all done on our Dev system, however. The Prod OS/config is a clone of the so I'm at a loss as to what the difference would be that's causing this to occur. ~Pete_Jibe Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains. -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
On 05/15/2013 02:24 PM, Pete Beebe wrote: Hi Thomas, The 4.0.11 to 4.0.12 Oracle upgrade completed and this RT At A Glance issue still did not reappear. This was all done on our Dev system, however. The Prod OS/config is a clone of the so I'm at a loss as to what the difference would be that's causing this to occur. That result is quite strange, but suggests a database inconsistency. Have you run rt-validator? http://bestpractical.com/rt/docs/latest/rt-validator What are the output of the 3 SQL queries I sent you earlier on this newly upgraded dev instance? As of yet, I haven't tracked down the problem locally. By my count, there are 3 reports of similar issues on rt-users. At least two of three migrated database types, so my suspicions are there right now but I have no proof/explanation. I hope to have more time to debug this soon. -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
[rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Your system config page and SQL results reveal that you have duplicate
copies (3 total of each record) of certain global data. This suggests
your migration from MySQL to Oracle was flawed. How did you migrate,
exactly?
As follows:
1. Created new OEL 6.4 x64 server running Apache/Mysqld from distrib packages.
Additionally installed a newer version of Perl and fought with character set
issues between old and new server configurations.
2. mysqldump'd data from old server and imported into an empty rt4 mysql
database on the new server
3. Upgraded mysql RT3 data to RT4 using make upgrade-database (4.0.12).
3. Prepped Oracle schema using RT 4.0.12 configuration and then truncated all
tables from new schema as well as disabled constraints as prep for import.
4. Ran a perl script obtained from a user from this thread:
http://requesttracker.8502.n7.nabble.com/mysql-to-oracle-migration-td34166.html.
Script pretty much copies records from their mysql tables into their
respective Oracle tables
5. Re-created the Oracle sequences after import.
6. Re-enabled Oracle table constraints.
I could forward you a PDF of the actual upgrade steps however it's not for
public digestion since it's replete with confidential data that'd be a PIA to
scrub.
What do these queries return?
select id, Name, Created, LastUpdated from Users where Name in ('RT_System',
'Nobody', 'root');
ID NAME,CREATED,LASTUPDATED
10 Nobody,6-Sep-05,6-Sep-05
1 RT_System,6-Sep-05,6-Sep-05
12 root,6-Sep-05, 4-May-13
select id, Domain, Type, Created, LastUpdated from Groups where Type in
('Everyone', 'Privileged', 'Unprivileged');
ID DOMAIN, TYPE,CREATED,LASTUPDATED
3 SystemInternal,Everyone,,
4 SystemInternal,Privileged,,
5 SystemInternal,Unprivileged,,
Disclaimer: This electronic message may contain information that is
Confidential or legally privileged. It is intended only for the use of the
individual(s) and entity named in the message. If you are not an intended
recipient of this message, please notify the sender immediately and delete the
material from your computer. Do not deliver, distribute or copy this message
and do not disclose its contents or take any action in reliance on the
information it contains.
--
RT Training in Seattle, June 19-20: http://bestpractical.com/training
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Thanks for the information, notes below.
On 05/09/2013 09:29 AM, Pete Beebe wrote:
Do you have any local customizations?
None that I'm aware of. Only plugins I've currently go in place are:
RT::Authen::ExternalAuth RT::Extension::ActivityReports
Can you send us the output of running this SQL query? select
id, Name, Description, ObjectType, ObjectId from Attributes
where ObjectType='RT::System';
Could you also navigate to Configuration - Tools - System
Config in RT, save that page as HTML, and send the file? Make
sure any sensitive data is redacted first.
Both the Select query and the system config HTML page and files have
been included in the attached ZIP file.
Your system config page and SQL results reveal that you have duplicate
copies (3 total of each record) of certain global data. This suggests
your migration from MySQL to Oracle was flawed. How did you migrate,
exactly?
Your database may contain duplicate internal records in other areas too,
which are likely to lead to more problems (and potentially very subtle
ones). What do these queries return?
select id, Name, Created, LastUpdated from Users where Name in
('RT_System', 'Nobody', 'root');
select id, Domain, Type, Created, LastUpdated from Groups where Type in
('Everyone', 'Privileged', 'Unprivileged');
I'm unsure of how to reply to this and have it also update the forum
thread so I've omitted the rt-users@lists.bestpractical.com address
when sending the e-mail.
You reply by sending to rt-users@lists.bestpractical.com. That sends to
the mailing list, and I'll read it since I'm subscribed.
--
RT Training in Seattle, June 19-20: http://bestpractical.com/training
Re: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
On 05/04/2013 07:22 PM, Pete Beebe wrote: I just completed a migration from MySQL to Oracle as well as an upgrade from 3.8.5 to 4.0.12. So far everything is running well however I have run across one error that occurs when, as super user, I click on the Global--RT At A Glance link. I receive an error in the WebUI followed by the resulting log file error message: [Sun May 5 02:14:09 2013] [error]: Can't call method Content on an undefined value at /rt4/share/html/Admin/Global/MyRT.html line 69, GEN38 line 445. Stack: [/rt4/share/html/Admin/Global/MyRT.html:69] [/rt4/share/html/Admin/autohandler:49] [/rt4/sbin/../lib/RT/Interface/Web.pm:634] [/rt4/sbin/../lib/RT/Interface/Web.pm:335] [/rt4/share/html/autohandler:53] (/rt4/sbin/../lib/RT/Interface/Web/Handler.pm:208) I can't reproduce this problem and looking at the code don't see how it would happen. Do you have any local customizations? Can you send us the output of running this SQL query? select id, Name, Description, ObjectType, ObjectId from Attributes where ObjectType='RT::System'; Could you also navigate to Configuration - Tools - System Config in RT, save that page as HTML, and send the file? Make sure any sensitive data is redacted first. -- RT Training in Seattle, June 19-20: http://bestpractical.com/training
[rt-users] Post upgrade 3.8.5 to 4.0.12 Global-RT At A Glance error
Hello, I just completed a migration from MySQL to Oracle as well as an upgrade from 3.8.5 to 4.0.12. So far everything is running well however I have run across one error that occurs when, as super user, I click on the Global--RT At A Glance link. I receive an error in the WebUI followed by the resulting log file error message: [Sun May 5 02:14:09 2013] [error]: Can't call method Content on an undefined value at /rt4/share/html/Admin/Global/MyRT.html line 69, GEN38 line 445. Stack: [/rt4/share/html/Admin/Global/MyRT.html:69] [/rt4/share/html/Admin/autohandler:49] [/rt4/sbin/../lib/RT/Interface/Web.pm:634] [/rt4/sbin/../lib/RT/Interface/Web.pm:335] [/rt4/share/html/autohandler:53] (/rt4/sbin/../lib/RT/Interface/Web/Handler.pm:208) This is running on OEL 6.4 x64, Oracle 11gR2 DB, httpd v.2.2.15 and perl v.5.10.1. Any help on where I can begin troubleshooting this would be wonderful. Cheers, ~Pete Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.
