Tried Centos last night, and no difference at all.
-------------------------------------------------
Justin Hayes
OpenBet Support Manager
justin.ha...@openbet.com
On 6 Sep 2010, at 20:49, Justin Hayes wrote:
> Hi Ruslan,
>
> Sorry looks like I shrunk the image too much. The thing I find odd is that
> there are others with similar hardware who don't get the problem. It'll be
> great if 3.10 fixes it for me, but I'd love to get to the bottom of it first.
> I'm pretty much positive it's not a DB issue, as I've tried different sizes
> of DB, tried postgres AND mysql etc. I don't think it's apache as I've tried
> the built in webserver with RT and no change there either.
>
> Currently trying to install RT on Centos given that Roy (who has kindly been
> helping me with details of his own setup) appears to have none of the same
> problems on that OS. Perhaps perl is just slow on the 64bit ubuntu we've
> currently got live.
>
> No idea if it's going to have any effect though :(
>
> Justin
>
> -------------------------------------------------
> Justin Hayes
> OpenBet Support Manager
> justin.ha...@openbet.com
>
> On 6 Sep 2010, at 18:37, Ruslan Zakirov wrote:
>
>> Justin.
>>
>> First of all, I can not read from the chart, but anyway history rendering
>> has been worked on in a new code branch. Probably this code will be part of
>> RT 3.10. Code at the moment is unstable, but eventually it wil be faster
>> then the current version.
>>
>> On Mon, Sep 6, 2010 at 8:56 PM, Justin Hayes <justin.ha...@openbet.com>
>> wrote:
>> So far we've tried installing RT on different hardware, both 32 and 64bit
>> versions of linux. RT is still very slow for long tickets. All the time is
>> taken up by the perl/apache process maxing out a core of CPU.
>>
>> We've even gone as far as trying to profile the code. We came up with this
>> graph of where the time was going:
>>
>> <TIMING.png>
>> We then tried to go further into those functions but can't find a single
>> smoking gun call that is taking all the time.
>>
>> For example in a ticket that takes 22s to render approx 5 secs goes on these
>> 2 lines:
>>
>> File: Ticket/Elements/ShowHistory line: 100-103 version 3.8.8
>>
>> my @trans_attachments = grep { $_->TransactionId == $Transaction->Id }
>> @attachments;
>>
>> grep { ($_->TransactionId == $Transaction->Id ) &&
>> ($trans_content->{$_->Id} = $_) } @attachment_content;
>>
>> Both are greps. Does this imply that perl itself is just slow?
>>
>> IF so why would our perl be slow compared to other people's? We've tried
>> compiling it from source and that made no difference.
>>
>> ATM we're at a bit of a loss....
>>
>> Justin
>>
>> -------------------------------------------------
>> Justin Hayes
>> OpenBet Support Manager
>> justin.ha...@openbet.com
>>
>> On 1 Jul 2010, at 11:51, Raed El-Hames wrote:
>>
>>> Justin,
>>>
>>> Do you use Transaction custom fields, if you do n’t ; try and comment out
>>> lines 70,71,72 from html/Ticket/Elements/ShowTransaction
>>> % if ( $Transaction->CustomFieldValues->Count ) {
>>> <& /Elements/ShowCustomFields, Object => $Transaction &>
>>> % }
>>> See if that improves things for you.
>>> Some of our monitoring tickets can have up to 500 updates, such tickets use
>>> to take up to 20s to load, once I commented out the above lines, load time
>>> is now down to less than 5 seconds.
>>>
>>> Regards;
>>> Roy
>>>
>>>
>>> From: rt-users-boun...@lists.bestpractical.com
>>> [mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Justin Hayes
>>>
>>> Sent: 01 July 2010 11:39
>>> To: Kenneth Crocker
>>> Cc: rt-users@lists.bestpractical.com
>>> Subject: Re: [rt-users] Slow Ticket History 3.8.8
>>>
>>> We do Kenneth, but most tickets don't have many file attachments, so I
>>> assume that's not an issue?
>>>
>>> Cheers,
>>>
>>> Justin
>>>
>>> -------------------------------------------------
>>> Justin Hayes
>>> OpenBet Support Manager
>>> justin.ha...@openbet.com
>>>
>>> On 29 Jun 2010, at 17:54, Kenneth Crocker wrote:
>>>
>>>
>>> Justin,
>>>
>>> I didn't see this mentioned and may have missed it, but are you displaying
>>> attachements inline? That might cut back on the I/O for History. Just a
>>> thought.
>>>
>>> Kenn
>>> LBNL
>>>
>>> On Tue, Jun 29, 2010 at 8:04 AM, Justin Hayes <justin.ha...@openbet.com>
>>> wrote:
>>> As a test we've just created a long ticket in an empty RT DB and it's very
>>> fast. So does look to be DB related - contrary to our earlier
>>> investigations.
>>>
>>> I guess it must still access the DB resultset during the ticket rendering
>>> (which isn't how we thought it would work).
>>>
>>> Time to tune the hell out of mysql then.......
>>>
>>> Justin
>>>
>>> -------------------------------------------------
>>> Justin Hayes
>>> OpenBet Support Manager
>>> justin.ha...@openbet.com
>>>
>>> On 29 Jun 2010, at 15:53, Justin Hayes wrote:
>>>
>>> > Seem to be quite a few things to look at Jason. Need to figure out what
>>> > they all mean first.
>>> >
>>> > Justin
>>> >
>>> > -------- General Statistics
>>> > --------------------------------------------------
>>> > [--] Skipped version check for MySQLTuner script
>>> > [OK] Currently running supported MySQL version 5.1.37-1ubuntu5.4-log
>>> > [OK] Operating on 64-bit architecture
>>> >
>>> > -------- Storage Engine Statistics
>>> > -------------------------------------------
>>> > [--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
>>> > [--] Data in MyISAM tables: 611M (Tables: 8)
>>> > [--] Data in InnoDB tables: 10G (Tables: 20)
>>> > [!!] Total fragmented tables: 21
>>> >
>>> > -------- Performance Metrics
>>> > -------------------------------------------------
>>> > [--] Up for: 19d 19h 32m 37s (110M q [64.266 qps], 222K conn, TX: 637B,
>>> > RX: 39B)
>>> > [--] Reads / Writes: 98% / 2%
>>> > [--] Total buffers: 602.0M global + 134.8M per thread (150 max threads)
>>> > [!!] Maximum possible memory usage: 20.3G (262% of installed RAM)
>>> > [OK] Slow queries: 0% (229K/110M)
>>> > [!!] Highest connection usage: 100% (151/150)
>>> > [OK] Key buffer size / total MyISAM indexes: 512.0M/6.7M
>>> > [OK] Key buffer hit rate: 100.0% (84M cached / 7K reads)
>>> > [OK] Query cache efficiency: 71.4% (76M cached / 107M selects)
>>> > [!!] Query cache prunes per day: 661360
>>> > [OK] Sorts requiring temporary tables: 0% (0 temp sorts / 2M sorts)
>>> > [!!] Joins performed without indexes: 112714
>>> > [!!] Temporary tables created on disk: 33% (968K on disk / 2M total)
>>> > [OK] Thread cache hit rate: 99% (1K created / 222K connections)
>>> > [OK] Table cache hit rate: 36% (318 open / 880 opened)
>>> > [OK] Open file limit used: 14% (166/1K)
>>> > [OK] Table locks acquired immediately: 99% (39M immediate / 39M locks)
>>> > [!!] InnoDB data size / buffer pool: 10.1G/8.0M
>>> >
>>> > -------- Recommendations
>>> > -----------------------------------------------------
>>> > General recommendations:
>>> > Run OPTIMIZE TABLE to defragment tables for better performance
>>> > Reduce your overall MySQL memory footprint for system stability
>>> > Reduce or eliminate persistent connections to reduce connection usage
>>> > Adjust your join queries to always utilize indexes
>>> > When making adjustments, make tmp_table_size/max_heap_table_size equal
>>> > Reduce your SELECT DISTINCT queries without LIMIT clauses
>>> > Variables to adjust:
>>> > *** MySQL's maximum memory usage is dangerously high ***
>>> > *** Add RAM before increasing MySQL buffer variables ***
>>> > max_connections (> 150)
>>> > wait_timeout (< 28800)
>>> > interactive_timeout (< 28800)
>>> > query_cache_size (> 16M)
>>> > join_buffer_size (> 2.0M, or always use indexes with joins)
>>> > tmp_table_size (> 128M)
>>> > max_heap_table_size (> 64M)
>>> > innodb_buffer_pool_size (>= 10G)
>>> >
>>> >
>>> > -------------------------------------------------
>>> > Justin Hayes
>>> > OpenBet Support Manager
>>> > justin.ha...@openbet.com
>>> >
>>> > On 29 Jun 2010, at 15:22, Jason Doran wrote:
>>> >
>>> >> Hi,
>>> >> If you are using mysqld have a look at "mysqltuner.pl" perl script
>>> >> (google)
>>> >> This has fixed quickly many performance issues on both RT and other
>>> >> web-based software we use. I run this every few weeks and apply suggested
>>> >> changes and then simply restart mysqld when things are quite.
>>> >>
>>> >> Regards,
>>> >> Jason Doran
>>> >> Computer Centre
>>> >> NUI, Maynooth
>>> >>
>>> >> On 29 Jun 2010, at 14:09, Justin Hayes wrote:
>>> >>
>>> >>> Hi everyone,
>>> >>>
>>> >>> I've raised this before, but we've had another look at it and still
>>> >>> can't see how to improve things.
>>> >>>
>>> >>> We put a lot of comments/replies in our tickets. Often there can be
>>> >>> 50-100 entries in a ticket, mostly plain text. Loading such a ticket
>>> >>> can take 10-20secs.
>>> >>>
>>> >>> We don't have any slow queries - all the time seems to be in the code
>>> >>> rendering the history of the ticket.
>>> >>> We've had a go at stripping functions out of ShowHistory,
>>> >>> ShowTransaction and ShowTransactionAttachmments but not had much
>>> >>> success.
>>> >>>
>>> >>> FWIW our RT runs on quad 3ghz Xeons with 8gb of ram.
>>> >>>
>>> >>> I'd like to try and determine if we're just slow, or if this is just
>>> >>> how long RT takes. Maybe perl is just slow.
>>> >>>
>>> >>> Can anyone shed any light on how long it takes them to render long
>>> >>> tickets in their systems? If you look at the page source it gives you a
>>> >>> value e.g.
>>> >>>
>>> >>> <span>Time to display: 24.996907</span>
>>> >>>
>>> >>> Can anyone share some numbers from theirs for longer tickets? It would
>>> >>> be really appreciated.
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Justin
>>> >>>
>>> >>> -------------------------------------------------
>>> >>> Justin Hayes
>>> >>> OpenBet Support Manager
>>> >>> justin.ha...@openbet.com
>>> >>>
>>> >>>
>>> >>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>>> >>> Buy a copy at http://rtbook.bestpractical.com
>>> >>
>>> >
>>> >
>>> > Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>>> > Buy a copy at http://rtbook.bestpractical.com
>>>
>>>
>>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>>> Buy a copy at http://rtbook.bestpractical.com
>>>
>>>
>>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>>> Buy a copy at http://rtbook.bestpractical.com
>>>
>>
>>
>>
>> RT Training in Washington DC, USA on Oct 25 & 26 2010
>> Last one this year -- Learn how to get the most out of RT!
>>
>>
>>
>> --
>> Best regards, Ruslan.
>
>
> RT Training in Washington DC, USA on Oct 25 & 26 2010
> Last one this year -- Learn how to get the most out of RT!
RT Training in Washington DC, USA on Oct 25 & 26 2010
Last one this year -- Learn how to get the most out of RT!
