Alfred von Campe wrote:
We are experiencing a strange hang when viewing one particular review.  When we click on the 
"View DIff" we get a new web page and the "Files Changed:" section lists all 
the files that have changed in this particular request.  There are 20 files, and the information 
for the first 6 is updated in a few seconds.  But then it hangs on the 7th file until we navigate 
away from the page (my last attempt was about 20 minutes ago and I still have 14 spinning circles 
on the screen).

I know I should upgrade to the 1.5.3 release from earlier today, but before I 
do that I want to try to debug this issue.  The problem is, I don't know where 
to even start.  BTW, when the user that stumbled across this issue reported it 
to me last week, I was able to load this particular review, but today it is 
hanging for me as well.  Always on the same file.  Funny thing is that there is 
only a very minor change to this file.  (I've tried multiple browsers in case 
that matters).

I've had 2 main issues with "slow" diffs:

  1. database locking (for testing I was using sqlite which is single
     user and horrible for *large* [10Mb diffs] reviews as multi user
     concurrency is awful, for most reviews sqlite is actually pretty
     good).
  2. "complex" diffs being slow to generate diffs, this is a feature of
     the batteries includes difflib in python.

#1 is pretty easy to check, if you get timeouts and errors mailed to the admin about database locks you know you hit #1 :-)

#2 is more sinister as it doesn't happen that often but you can get pathologically "bad" diffs that takes mins/hours for difflib to examine. The only way to know if this is what you are hitting is to write a python script to use difflib with the original and new file.

I've attached a demo of #2 where there is a subtle difference mid-line for each line. There is no easy way around this unless a different diff module is used.

NOTE attached demo uses an iterator, if you do NOT use an iterator it still takes 2 mins but you get no output during that 2 minute period and so looks like it has hung!

I've no idea if you are hitting these but I've been meaning to post my experiences in this area for some time and this was a good reminder.

Chris

"""demo of built in diff lib cpu clobbering
really heavy cpu intensive, but performs VERY SLOWLY (it looks like it hangs, 
but it does complete)
"""

import difflib

file1content =  ['27 mss gw001 PASS\n', '27 mss gw002 PASS\n', '27 mss gw003 
PASS\n', '27 mss gw004 PASS\n', '27 mss gw005 PASS\n', '27 mss gw006 PASS\n', 
'27 mss gw007 PASS\n', '27 mss gw008 PASS\n', '27 mss gw009 PASS\n', '27 mss 
gw010 PASS\n', '27 mss gw011 PASS\n', '27 mss gw012 PASS\n', '27 mss gw013 
PASS\n', '27 mss gw014 PASS\n', '27 mss gw015 PASS\n', '27 mss gw016 PASS\n', 
'27 mss gw017 FAIL\n', '27 mss gw018 PASS\n', '27 mss gw019 PASS\n', '27 mss 
gw020 PASS\n', '27 mss gw021 FAIL\n', '27 mss gw022 PASS\n', '27 mss gw023 
PASS\n', '27 mss gw024 PASS\n', '27 mss gw025 PASS\n', '27 mss gw026 PASS\n', 
'27 mss gw027 PASS\n', '27 mss gw028 PASS\n', '27 mss gw029 PASS\n', '27 mss 
gw030 PASS\n', '27 mss gw031 FAIL\n', '27 mss gw032 PASS\n', '27 mss gw033 
FAIL\n', '27 mss gw034 PASS\n', '27 mss gw035 PASS\n', '27 mss gw036 PASS\n', 
'27 mss gw037 PASS\n', '27 mss gw038 PASS\n', '27 mss gw039 FAIL\n', '27 mss 
gw040 FAIL\n', '27 mss gw041 PASS\n', '27 mss gw042 PASS\n', '27 mss gw043 
PASS\n', '27 mss gw044 PASS\n', '27 mss gw045 PASS\n', '27 mss gw046 PASS\n', 
'27 mss gw047 PASS\n', '27 mss gw048 PASS\n', '27 mss gw049 PASS\n', '27 mss 
gw050 PASS\n', '27 mss gw051 PASS\n', '27 mss gw052 PASS\n', '27 mss gw053 
PASS\n', '27 mss gw054 PASS\n', '27 mss gw055 PASS\n', '27 mss gw056 PASS\n', 
'27 mss gw057 PASS\n', '27 mss gw058 PASS\n', '27 mss gw059 PASS\n', '27 mss 
gw060 PASS\n', '27 mss gw061 PASS\n', '27 mss gw062 PASS\n', '27 mss gw063 
PASS\n', '27 mss gw064 PASS\n', '27 mss gw065 PASS\n', '27 mss gw066 PASS\n', 
'27 mss gw067 PASS\n', '27 mss gw068 PASS\n', '27 mss gw071 PASS\n', '27 mss 
gw075 PASS\n', '27 mss gw076 PASS\n', '27 mss gw077 PASS\n', '27 mss gw078 
PASS\n', '27 mss gw079 PASS\n', '27 mss gw081 PASS\n', '27 mss gw085 PASS\n', 
'27 mss gw086 PASS\n', '27 mss gw088 PASS\n', '27 mss gw091 PASS\n', '27 mss 
gw092 PASS\n', '27 mss gw093 PASS\n', '27 mss gw094 PASS\n', '27 mss gw095 
PASS\n', '27 mss gw096 PASS\n', '27 mss gw097 PASS\n', '27 mss gw098 PASS\n', 
'27 mss gw099 PASS\n', '27 mss gw100 PASS\n', '27 mss gw101 PASS\n', '27 mss 
gw102 PASS\n', '27 mss gw103 PASS\n', '27 mss gw104 PASS\n', '27 mss gw105 
PASS\n', '27 mss gw106 PASS\n', '27 mss gw108 PASS\n', '27 mss gw109 PASS\n', 
'27 mss gw110 PASS\n', '27 mss gw111 PASS\n', '27 mss gw112 PASS\n', '27 mss 
gw113 PASS\n', '27 mss gw114 PASS\n', '27 mss gw115 PASS\n', '27 mss gw116 
PASS\n', '27 mss gw117 PASS\n', '27 mss gw118 PASS\n', '27 mss gw120 PASS\n', 
'27 mss gw121 PASS\n', '27 mss gw122 PASS\n', '27 mss gw123 PASS\n', '27 mss 
gw124 PASS\n', '27 mss gw125 PASS\n', '27 mss gw150 PASS\n', '27 mss gw153 
PASS\n', '27 mss gw155 FAIL\n', '27 mss gw165 PASS\n', '27 mss gw168 PASS\n', 
'27 mss gw169 PASS\n', '27 mss gw170 PASS\n', '27 mss gw171 PASS\n', '27 mss 
gw172 PASS\n', '27 mss gw174 PASS\n', '27 mss gw175 PASS\n', '27 mss gw176 
PASS\n', '27 mss gw178 PASS\n', '27 mss gw179 PASS\n', '27 mss gw180 PASS\n', 
'27 mss gw181 PASS\n', '27 mss gw182 PASS\n', '27 mss gw183 PASS\n', '27 mss 
gw184 PASS\n', '27 mss gw185 PASS\n', '27 mss gw186 PASS\n', '27 mss gw187 
PASS\n', '27 mss gw188 PASS\n', '27 mss gw189 PASS\n', '27 mss gw190 FAIL\n', 
'27 mss gw191 PASS\n', '27 mss gw192 PASS\n', '27 mss gw193 PASS\n', '27 mss 
gw194 PASS\n', '27 mss gw195 PASS\n', '27 mss gw196 FAIL\n', '27 mss gw200 
PASS\n', '27 mss gw201 PASS\n', '27 mss gw202 PASS\n', '27 mss gw203 PASS\n', 
'27 mss gw204 PASS\n', '27 mss gw205 PASS\n', '27 mss gw206 PASS\n', '27 mss 
gw207 PASS\n', '27 mss gw209 PASS\n', '27 mss gw210 PASS\n', '27 mss gw231 
FAIL\n', '27 mss gw232 FAIL\n', '27 mss gw260 PASS\n', '27 mss gw320 PASS\n', 
'27 mss gw321 FAIL\n', '27 mss gw322 PASS\n', '27 mss gw400 PASS\n', '27 mss 
gw401 PASS\n', '27 mss gw402 FAIL\n', '27 mss gw403 PASS\n', '27 mss gw404 
PASS\n', '27 mss gw405 PASS\n', '27 mss gw406 PASS\n', '27 mss gw440 PASS\n', 
'27 mss gw441 PASS\n', '27 mss gw442 PASS\n', '27 mss gw443 PASS\n', '27 mss 
gw444 PASS\n', '27 mss gw445 PASS\n', '27 mss gw446 PASS\n', '27 mss gw500 
PASS\n', '27 mss gw501 PASS\n', '27 mss gw503 FAIL\n', '27 mss gw504 PASS\n', 
'27 mss gw505 PASS\n', '27 mss gw506 PASS\n', '27 mss gw507 PASS\n', '27 mss 
gw508 PASS\n', '27 mss gw509 PASS\n', '27 mss gw510 FAIL\n', '27 mss gw511 
PASS\n', '27 mss gw512 PASS\n', '27 mss gw513 PASS\n', '27 mss gw514 PASS\n', 
'27 mss gw515 PASS\n', '27 mss gw516 PASS\n', '27 mss gw517 FAIL\n', '27 mss 
gw530 PASS\n', '27 mss gw532 PASS\n', '27 mss gw584 PASS\n', '27 mss gw585 
PASS\n', '27 mss gw586 PASS\n', '27 mss gw590 PASS\n', '27 mss gw591 PASS\n', 
'27 mss gw592 PASS\n', '27 mss gw593 PASS\n', '27 mss gw594 PASS\n', '27 mss 
gw596 PASS\n', '27 mss gw597 PASS\n', '27 mss gw598 PASS\n', '27 mss gw600 
PASS\n', '27 mss gw601 PASS\n', '27 mss gw604 PASS\n', '27 mss gw605 PASS\n', 
'27 mss gw606 FAIL\n', '27 mss gw607 PASS\n', '27 mss gw608 PASS\n', '27 mss 
gw609 PASS\n', '27 mss gw611 PASS\n', '27 mss gw612 PASS\n', '27 mss gw613 
PASS\n', '27 mss gw614 PASS\n', '27 mss gw615 PASS\n', '27 mss gw616 PASS\n', 
'27 mss gw650 PASS\n', '27 mss gw653 PASS\n', '27 mss gw655 PASS\n', '27 mss 
gw656 PASS\n', '27 mss gw657 PASS\n', '27 mss gw658 PASS\n', '27 mss gw659 
FAIL\n', '27 mss gw662 FAIL\n', '27 mss gw663 FAIL\n', '27 mss gw680 FAIL\n', 
'27 mss gw700 PASS\n', '27 mss gw701 PASS\n', '27 mss gw702 PASS\n', '27 mss 
gw703 PASS\n', '27 mss gw704 PASS\n', '27 mss gw705 PASS\n', '27 mss gw706 
PASS\n', '27 mss gw707 PASS\n', '27 mss gw708 PASS\n', '27 mss gw709 PASS\n', 
'27 mss gw710 PASS\n', '27 mss gw711 PASS\n', '27 mss gw713 PASS\n', '27 mss 
gw714 PASS\n', '27 mss gw715 PASS\n', '27 mss gw718 PASS\n', '27 mss gw900 
PASS\n', '27 mss gw988 PASS\n', '27 mss gw989 PASS\n', '27 mss gw992 PASS\n', 
'27 mss gw993 PASS\n', '27 mss gw994 PASS\n', '27 mss gw995 PASS\n', '27 mss 
gw996 PASS\n', '27 mss gw999 PASS\n']
file2content =  ['27 ora gw001 PASS\n', '27 ora gw002 PASS\n', '27 ora gw003 
PASS\n', '27 ora gw004 PASS\n', '27 ora gw005 PASS\n', '27 ora gw006 PASS\n', 
'27 ora gw007 PASS\n', '27 ora gw008 PASS\n', '27 ora gw009 PASS\n', '27 ora 
gw010 PASS\n', '27 ora gw011 PASS\n', '27 ora gw012 PASS\n', '27 ora gw013 
PASS\n', '27 ora gw014 PASS\n', '27 ora gw015 PASS\n', '27 ora gw016 PASS\n', 
'27 ora gw017 FAIL\n', '27 ora gw018 PASS\n', '27 ora gw019 PASS\n', '27 ora 
gw020 PASS\n', '27 ora gw021 PASS\n', '27 ora gw022 PASS\n', '27 ora gw023 
PASS\n', '27 ora gw024 PASS\n', '27 ora gw025 PASS\n', '27 ora gw026 PASS\n', 
'27 ora gw027 PASS\n', '27 ora gw028 PASS\n', '27 ora gw029 PASS\n', '27 ora 
gw030 PASS\n', '27 ora gw031 PASS\n', '27 ora gw032 PASS\n', '27 ora gw033 
PASS\n', '27 ora gw034 PASS\n', '27 ora gw035 PASS\n', '27 ora gw036 PASS\n', 
'27 ora gw037 PASS\n', '27 ora gw038 PASS\n', '27 ora gw039 PASS\n', '27 ora 
gw040 PASS\n', '27 ora gw041 PASS\n', '27 ora gw042 PASS\n', '27 ora gw043 
PASS\n', '27 ora gw044 PASS\n', '27 ora gw045 PASS\n', '27 ora gw046 PASS\n', 
'27 ora gw047 PASS\n', '27 ora gw048 PASS\n', '27 ora gw049 PASS\n', '27 ora 
gw050 PASS\n', '27 ora gw051 PASS\n', '27 ora gw052 PASS\n', '27 ora gw053 
PASS\n', '27 ora gw054 PASS\n', '27 ora gw055 PASS\n', '27 ora gw056 PASS\n', 
'27 ora gw057 PASS\n', '27 ora gw058 PASS\n', '27 ora gw059 PASS\n', '27 ora 
gw060 PASS\n', '27 ora gw061 PASS\n', '27 ora gw062 PASS\n', '27 ora gw063 
PASS\n', '27 ora gw064 PASS\n', '27 ora gw065 PASS\n', '27 ora gw066 PASS\n', 
'27 ora gw067 PASS\n', '27 ora gw068 PASS\n', '27 ora gw071 PASS\n', '27 ora 
gw075 PASS\n', '27 ora gw076 PASS\n', '27 ora gw077 PASS\n', '27 ora gw078 
PASS\n', '27 ora gw079 PASS\n', '27 ora gw081 PASS\n', '27 ora gw085 PASS\n', 
'27 ora gw086 PASS\n', '27 ora gw088 PASS\n', '27 ora gw091 PASS\n', '27 ora 
gw092 PASS\n', '27 ora gw093 PASS\n', '27 ora gw094 PASS\n', '27 ora gw095 
PASS\n', '27 ora gw096 PASS\n', '27 ora gw097 PASS\n', '27 ora gw098 PASS\n', 
'27 ora gw099 PASS\n', '27 ora gw100 PASS\n', '27 ora gw101 PASS\n', '27 ora 
gw102 PASS\n', '27 ora gw103 PASS\n', '27 ora gw104 PASS\n', '27 ora gw105 
PASS\n', '27 ora gw106 PASS\n', '27 ora gw108 PASS\n', '27 ora gw109 PASS\n', 
'27 ora gw110 PASS\n', '27 ora gw111 PASS\n', '27 ora gw112 PASS\n', '27 ora 
gw113 PASS\n', '27 ora gw114 PASS\n', '27 ora gw115 PASS\n', '27 ora gw116 
PASS\n', '27 ora gw117 PASS\n', '27 ora gw118 PASS\n', '27 ora gw120 PASS\n', 
'27 ora gw121 PASS\n', '27 ora gw122 PASS\n', '27 ora gw123 PASS\n', '27 ora 
gw124 PASS\n', '27 ora gw125 PASS\n', '27 ora gw150 PASS\n', '27 ora gw153 
PASS\n', '27 ora gw155 FAIL\n', '27 ora gw156 PASS\n', '27 ora gw165 PASS\n', 
'27 ora gw168 PASS\n', '27 ora gw169 PASS\n', '27 ora gw170 PASS\n', '27 ora 
gw171 PASS\n', '27 ora gw172 PASS\n', '27 ora gw174 PASS\n', '27 ora gw175 
PASS\n', '27 ora gw176 PASS\n', '27 ora gw178 PASS\n', '27 ora gw179 PASS\n', 
'27 ora gw180 PASS\n', '27 ora gw181 PASS\n', '27 ora gw182 PASS\n', '27 ora 
gw183 PASS\n', '27 ora gw184 PASS\n', '27 ora gw185 PASS\n', '27 ora gw186 
PASS\n', '27 ora gw187 PASS\n', '27 ora gw188 PASS\n', '27 ora gw189 PASS\n', 
'27 ora gw190 PASS\n', '27 ora gw191 PASS\n', '27 ora gw192 PASS\n', '27 ora 
gw193 PASS\n', '27 ora gw194 PASS\n', '27 ora gw195 PASS\n', '27 ora gw196 
FAIL\n', '27 ora gw200 PASS\n', '27 ora gw201 PASS\n', '27 ora gw202 PASS\n', 
'27 ora gw203 PASS\n', '27 ora gw204 PASS\n', '27 ora gw205 PASS\n', '27 ora 
gw206 PASS\n', '27 ora gw207 PASS\n', '27 ora gw209 PASS\n', '27 ora gw210 
PASS\n', '27 ora gw231 PASS\n', '27 ora gw232 PASS\n', '27 ora gw260 PASS\n', 
'27 ora gw320 PASS\n', '27 ora gw321 PASS\n', '27 ora gw322 PASS\n', '27 ora 
gw400 PASS\n', '27 ora gw401 PASS\n', '27 ora gw402 FAIL\n', '27 ora gw403 
PASS\n', '27 ora gw404 PASS\n', '27 ora gw405 PASS\n', '27 ora gw406 PASS\n', 
'27 ora gw500 PASS\n', '27 ora gw501 PASS\n', '27 ora gw503 PASS\n', '27 ora 
gw504 PASS\n', '27 ora gw505 PASS\n', '27 ora gw506 FAIL\n', '27 ora gw507 
PASS\n', '27 ora gw508 PASS\n', '27 ora gw509 PASS\n', '27 ora gw510 PASS\n', 
'27 ora gw511 PASS\n', '27 ora gw512 PASS\n', '27 ora gw513 PASS\n', '27 ora 
gw514 PASS\n', '27 ora gw515 PASS\n', '27 ora gw516 PASS\n', '27 ora gw517 
FAIL\n', '27 ora gw530 PASS\n', '27 ora gw532 PASS\n', '27 ora gw584 PASS\n', 
'27 ora gw585 PASS\n', '27 ora gw586 PASS\n', '27 ora gw590 PASS\n', '27 ora 
gw591 PASS\n', '27 ora gw592 PASS\n', '27 ora gw593 PASS\n', '27 ora gw594 
PASS\n', '27 ora gw596 PASS\n', '27 ora gw597 PASS\n', '27 ora gw598 PASS\n', 
'27 ora gw600 PASS\n', '27 ora gw601 PASS\n', '27 ora gw604 PASS\n', '27 ora 
gw605 PASS\n', '27 ora gw606 PASS\n', '27 ora gw607 PASS\n', '27 ora gw608 
PASS\n', '27 ora gw609 PASS\n', '27 ora gw611 PASS\n', '27 ora gw612 PASS\n', 
'27 ora gw613 PASS\n', '27 ora gw614 PASS\n', '27 ora gw615 PASS\n', '27 ora 
gw616 PASS\n', '27 ora gw650 PASS\n', '27 ora gw651 PASS\n', '27 ora gw653 
PASS\n', '27 ora gw655 PASS\n', '27 ora gw656 PASS\n', '27 ora gw657 PASS\n', 
'27 ora gw658 PASS\n', '27 ora gw659 FAIL\n', '27 ora gw662 PASS\n', '27 ora 
gw663 FAIL\n', '27 ora gw680 PASS\n', '27 ora gw700 PASS\n', '27 ora gw701 
PASS\n', '27 ora gw702 PASS\n', '27 ora gw703 PASS\n', '27 ora gw704 PASS\n', 
'27 ora gw705 FAIL\n', '27 ora gw706 PASS\n', '27 ora gw707 PASS\n', '27 ora 
gw708 PASS\n', '27 ora gw709 PASS\n', '27 ora gw710 PASS\n', '27 ora gw711 
PASS\n', '27 ora gw713 PASS\n', '27 ora gw714 PASS\n', '27 ora gw715 PASS\n', 
'27 ora gw718 PASS\n', '27 ora gw720 FAIL\n', '27 ora gw721 PASS\n', '27 ora 
gw722 FAIL\n', '27 ora gw900 FAIL\n', '27 ora gw988 PASS\n', '27 ora gw989 
PASS\n', '27 ora gw992 PASS\n', '27 ora gw993 PASS\n', '27 ora gw994 PASS\n', 
'27 ora gw995 PASS\n', '27 ora gw996 PASS\n', '27 ora gw999 PASS\n']

print('difflib.Differ()')
d = difflib.Differ()

#diffs = list(d.compare(file1content, file2content))

print('d.compare(file1content, file2content)')
diffs = d.compare(file1content, file2content)

print(type(diffs))


print('for x in diffs: ## takes 2 mins!')
for x in diffs:
    print 'x', x
#print('list(diffs)')
#diffs = list(diffs)
result = ''.join(diffs)
print(result)
-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~----------~----~----~----~------~----~------~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en

Reply via email to