Alfred von Campe wrote:
We are experiencing a strange hang when viewing one particular review. When we click on the
"View DIff" we get a new web page and the "Files Changed:" section lists all
the files that have changed in this particular request. There are 20 files, and the information
for the first 6 is updated in a few seconds. But then it hangs on the 7th file until we navigate
away from the page (my last attempt was about 20 minutes ago and I still have 14 spinning circles
on the screen).
I know I should upgrade to the 1.5.3 release from earlier today, but before I
do that I want to try to debug this issue. The problem is, I don't know where
to even start. BTW, when the user that stumbled across this issue reported it
to me last week, I was able to load this particular review, but today it is
hanging for me as well. Always on the same file. Funny thing is that there is
only a very minor change to this file. (I've tried multiple browsers in case
that matters).
I've had 2 main issues with "slow" diffs:
1. database locking (for testing I was using sqlite which is single
user and horrible for *large* [10Mb diffs] reviews as multi user
concurrency is awful, for most reviews sqlite is actually pretty
good).
2. "complex" diffs being slow to generate diffs, this is a feature of
the batteries includes difflib in python.
#1 is pretty easy to check, if you get timeouts and errors mailed to the
admin about database locks you know you hit #1 :-)
#2 is more sinister as it doesn't happen that often but you can get
pathologically "bad" diffs that takes mins/hours for difflib to examine.
The only way to know if this is what you are hitting is to write a
python script to use difflib with the original and new file.
I've attached a demo of #2 where there is a subtle difference mid-line
for each line. There is no easy way around this unless a different diff
module is used.
NOTE attached demo uses an iterator, if you do NOT use an iterator it
still takes 2 mins but you get no output during that 2 minute period and
so looks like it has hung!
I've no idea if you are hitting these but I've been meaning to post my
experiences in this area for some time and this was a good reminder.
Chris
"""demo of built in diff lib cpu clobbering
really heavy cpu intensive, but performs VERY SLOWLY (it looks like it hangs,
but it does complete)
"""
import difflib
file1content = ['27 mss gw001 PASS\n', '27 mss gw002 PASS\n', '27 mss gw003
PASS\n', '27 mss gw004 PASS\n', '27 mss gw005 PASS\n', '27 mss gw006 PASS\n',
'27 mss gw007 PASS\n', '27 mss gw008 PASS\n', '27 mss gw009 PASS\n', '27 mss
gw010 PASS\n', '27 mss gw011 PASS\n', '27 mss gw012 PASS\n', '27 mss gw013
PASS\n', '27 mss gw014 PASS\n', '27 mss gw015 PASS\n', '27 mss gw016 PASS\n',
'27 mss gw017 FAIL\n', '27 mss gw018 PASS\n', '27 mss gw019 PASS\n', '27 mss
gw020 PASS\n', '27 mss gw021 FAIL\n', '27 mss gw022 PASS\n', '27 mss gw023
PASS\n', '27 mss gw024 PASS\n', '27 mss gw025 PASS\n', '27 mss gw026 PASS\n',
'27 mss gw027 PASS\n', '27 mss gw028 PASS\n', '27 mss gw029 PASS\n', '27 mss
gw030 PASS\n', '27 mss gw031 FAIL\n', '27 mss gw032 PASS\n', '27 mss gw033
FAIL\n', '27 mss gw034 PASS\n', '27 mss gw035 PASS\n', '27 mss gw036 PASS\n',
'27 mss gw037 PASS\n', '27 mss gw038 PASS\n', '27 mss gw039 FAIL\n', '27 mss
gw040 FAIL\n', '27 mss gw041 PASS\n', '27 mss gw042 PASS\n', '27 mss gw043
PASS\n', '27 mss gw044 PASS\n', '27 mss gw045 PASS\n', '27 mss gw046 PASS\n',
'27 mss gw047 PASS\n', '27 mss gw048 PASS\n', '27 mss gw049 PASS\n', '27 mss
gw050 PASS\n', '27 mss gw051 PASS\n', '27 mss gw052 PASS\n', '27 mss gw053
PASS\n', '27 mss gw054 PASS\n', '27 mss gw055 PASS\n', '27 mss gw056 PASS\n',
'27 mss gw057 PASS\n', '27 mss gw058 PASS\n', '27 mss gw059 PASS\n', '27 mss
gw060 PASS\n', '27 mss gw061 PASS\n', '27 mss gw062 PASS\n', '27 mss gw063
PASS\n', '27 mss gw064 PASS\n', '27 mss gw065 PASS\n', '27 mss gw066 PASS\n',
'27 mss gw067 PASS\n', '27 mss gw068 PASS\n', '27 mss gw071 PASS\n', '27 mss
gw075 PASS\n', '27 mss gw076 PASS\n', '27 mss gw077 PASS\n', '27 mss gw078
PASS\n', '27 mss gw079 PASS\n', '27 mss gw081 PASS\n', '27 mss gw085 PASS\n',
'27 mss gw086 PASS\n', '27 mss gw088 PASS\n', '27 mss gw091 PASS\n', '27 mss
gw092 PASS\n', '27 mss gw093 PASS\n', '27 mss gw094 PASS\n', '27 mss gw095
PASS\n', '27 mss gw096 PASS\n', '27 mss gw097 PASS\n', '27 mss gw098 PASS\n',
'27 mss gw099 PASS\n', '27 mss gw100 PASS\n', '27 mss gw101 PASS\n', '27 mss
gw102 PASS\n', '27 mss gw103 PASS\n', '27 mss gw104 PASS\n', '27 mss gw105
PASS\n', '27 mss gw106 PASS\n', '27 mss gw108 PASS\n', '27 mss gw109 PASS\n',
'27 mss gw110 PASS\n', '27 mss gw111 PASS\n', '27 mss gw112 PASS\n', '27 mss
gw113 PASS\n', '27 mss gw114 PASS\n', '27 mss gw115 PASS\n', '27 mss gw116
PASS\n', '27 mss gw117 PASS\n', '27 mss gw118 PASS\n', '27 mss gw120 PASS\n',
'27 mss gw121 PASS\n', '27 mss gw122 PASS\n', '27 mss gw123 PASS\n', '27 mss
gw124 PASS\n', '27 mss gw125 PASS\n', '27 mss gw150 PASS\n', '27 mss gw153
PASS\n', '27 mss gw155 FAIL\n', '27 mss gw165 PASS\n', '27 mss gw168 PASS\n',
'27 mss gw169 PASS\n', '27 mss gw170 PASS\n', '27 mss gw171 PASS\n', '27 mss
gw172 PASS\n', '27 mss gw174 PASS\n', '27 mss gw175 PASS\n', '27 mss gw176
PASS\n', '27 mss gw178 PASS\n', '27 mss gw179 PASS\n', '27 mss gw180 PASS\n',
'27 mss gw181 PASS\n', '27 mss gw182 PASS\n', '27 mss gw183 PASS\n', '27 mss
gw184 PASS\n', '27 mss gw185 PASS\n', '27 mss gw186 PASS\n', '27 mss gw187
PASS\n', '27 mss gw188 PASS\n', '27 mss gw189 PASS\n', '27 mss gw190 FAIL\n',
'27 mss gw191 PASS\n', '27 mss gw192 PASS\n', '27 mss gw193 PASS\n', '27 mss
gw194 PASS\n', '27 mss gw195 PASS\n', '27 mss gw196 FAIL\n', '27 mss gw200
PASS\n', '27 mss gw201 PASS\n', '27 mss gw202 PASS\n', '27 mss gw203 PASS\n',
'27 mss gw204 PASS\n', '27 mss gw205 PASS\n', '27 mss gw206 PASS\n', '27 mss
gw207 PASS\n', '27 mss gw209 PASS\n', '27 mss gw210 PASS\n', '27 mss gw231
FAIL\n', '27 mss gw232 FAIL\n', '27 mss gw260 PASS\n', '27 mss gw320 PASS\n',
'27 mss gw321 FAIL\n', '27 mss gw322 PASS\n', '27 mss gw400 PASS\n', '27 mss
gw401 PASS\n', '27 mss gw402 FAIL\n', '27 mss gw403 PASS\n', '27 mss gw404
PASS\n', '27 mss gw405 PASS\n', '27 mss gw406 PASS\n', '27 mss gw440 PASS\n',
'27 mss gw441 PASS\n', '27 mss gw442 PASS\n', '27 mss gw443 PASS\n', '27 mss
gw444 PASS\n', '27 mss gw445 PASS\n', '27 mss gw446 PASS\n', '27 mss gw500
PASS\n', '27 mss gw501 PASS\n', '27 mss gw503 FAIL\n', '27 mss gw504 PASS\n',
'27 mss gw505 PASS\n', '27 mss gw506 PASS\n', '27 mss gw507 PASS\n', '27 mss
gw508 PASS\n', '27 mss gw509 PASS\n', '27 mss gw510 FAIL\n', '27 mss gw511
PASS\n', '27 mss gw512 PASS\n', '27 mss gw513 PASS\n', '27 mss gw514 PASS\n',
'27 mss gw515 PASS\n', '27 mss gw516 PASS\n', '27 mss gw517 FAIL\n', '27 mss
gw530 PASS\n', '27 mss gw532 PASS\n', '27 mss gw584 PASS\n', '27 mss gw585
PASS\n', '27 mss gw586 PASS\n', '27 mss gw590 PASS\n', '27 mss gw591 PASS\n',
'27 mss gw592 PASS\n', '27 mss gw593 PASS\n', '27 mss gw594 PASS\n', '27 mss
gw596 PASS\n', '27 mss gw597 PASS\n', '27 mss gw598 PASS\n', '27 mss gw600
PASS\n', '27 mss gw601 PASS\n', '27 mss gw604 PASS\n', '27 mss gw605 PASS\n',
'27 mss gw606 FAIL\n', '27 mss gw607 PASS\n', '27 mss gw608 PASS\n', '27 mss
gw609 PASS\n', '27 mss gw611 PASS\n', '27 mss gw612 PASS\n', '27 mss gw613
PASS\n', '27 mss gw614 PASS\n', '27 mss gw615 PASS\n', '27 mss gw616 PASS\n',
'27 mss gw650 PASS\n', '27 mss gw653 PASS\n', '27 mss gw655 PASS\n', '27 mss
gw656 PASS\n', '27 mss gw657 PASS\n', '27 mss gw658 PASS\n', '27 mss gw659
FAIL\n', '27 mss gw662 FAIL\n', '27 mss gw663 FAIL\n', '27 mss gw680 FAIL\n',
'27 mss gw700 PASS\n', '27 mss gw701 PASS\n', '27 mss gw702 PASS\n', '27 mss
gw703 PASS\n', '27 mss gw704 PASS\n', '27 mss gw705 PASS\n', '27 mss gw706
PASS\n', '27 mss gw707 PASS\n', '27 mss gw708 PASS\n', '27 mss gw709 PASS\n',
'27 mss gw710 PASS\n', '27 mss gw711 PASS\n', '27 mss gw713 PASS\n', '27 mss
gw714 PASS\n', '27 mss gw715 PASS\n', '27 mss gw718 PASS\n', '27 mss gw900
PASS\n', '27 mss gw988 PASS\n', '27 mss gw989 PASS\n', '27 mss gw992 PASS\n',
'27 mss gw993 PASS\n', '27 mss gw994 PASS\n', '27 mss gw995 PASS\n', '27 mss
gw996 PASS\n', '27 mss gw999 PASS\n']
file2content = ['27 ora gw001 PASS\n', '27 ora gw002 PASS\n', '27 ora gw003
PASS\n', '27 ora gw004 PASS\n', '27 ora gw005 PASS\n', '27 ora gw006 PASS\n',
'27 ora gw007 PASS\n', '27 ora gw008 PASS\n', '27 ora gw009 PASS\n', '27 ora
gw010 PASS\n', '27 ora gw011 PASS\n', '27 ora gw012 PASS\n', '27 ora gw013
PASS\n', '27 ora gw014 PASS\n', '27 ora gw015 PASS\n', '27 ora gw016 PASS\n',
'27 ora gw017 FAIL\n', '27 ora gw018 PASS\n', '27 ora gw019 PASS\n', '27 ora
gw020 PASS\n', '27 ora gw021 PASS\n', '27 ora gw022 PASS\n', '27 ora gw023
PASS\n', '27 ora gw024 PASS\n', '27 ora gw025 PASS\n', '27 ora gw026 PASS\n',
'27 ora gw027 PASS\n', '27 ora gw028 PASS\n', '27 ora gw029 PASS\n', '27 ora
gw030 PASS\n', '27 ora gw031 PASS\n', '27 ora gw032 PASS\n', '27 ora gw033
PASS\n', '27 ora gw034 PASS\n', '27 ora gw035 PASS\n', '27 ora gw036 PASS\n',
'27 ora gw037 PASS\n', '27 ora gw038 PASS\n', '27 ora gw039 PASS\n', '27 ora
gw040 PASS\n', '27 ora gw041 PASS\n', '27 ora gw042 PASS\n', '27 ora gw043
PASS\n', '27 ora gw044 PASS\n', '27 ora gw045 PASS\n', '27 ora gw046 PASS\n',
'27 ora gw047 PASS\n', '27 ora gw048 PASS\n', '27 ora gw049 PASS\n', '27 ora
gw050 PASS\n', '27 ora gw051 PASS\n', '27 ora gw052 PASS\n', '27 ora gw053
PASS\n', '27 ora gw054 PASS\n', '27 ora gw055 PASS\n', '27 ora gw056 PASS\n',
'27 ora gw057 PASS\n', '27 ora gw058 PASS\n', '27 ora gw059 PASS\n', '27 ora
gw060 PASS\n', '27 ora gw061 PASS\n', '27 ora gw062 PASS\n', '27 ora gw063
PASS\n', '27 ora gw064 PASS\n', '27 ora gw065 PASS\n', '27 ora gw066 PASS\n',
'27 ora gw067 PASS\n', '27 ora gw068 PASS\n', '27 ora gw071 PASS\n', '27 ora
gw075 PASS\n', '27 ora gw076 PASS\n', '27 ora gw077 PASS\n', '27 ora gw078
PASS\n', '27 ora gw079 PASS\n', '27 ora gw081 PASS\n', '27 ora gw085 PASS\n',
'27 ora gw086 PASS\n', '27 ora gw088 PASS\n', '27 ora gw091 PASS\n', '27 ora
gw092 PASS\n', '27 ora gw093 PASS\n', '27 ora gw094 PASS\n', '27 ora gw095
PASS\n', '27 ora gw096 PASS\n', '27 ora gw097 PASS\n', '27 ora gw098 PASS\n',
'27 ora gw099 PASS\n', '27 ora gw100 PASS\n', '27 ora gw101 PASS\n', '27 ora
gw102 PASS\n', '27 ora gw103 PASS\n', '27 ora gw104 PASS\n', '27 ora gw105
PASS\n', '27 ora gw106 PASS\n', '27 ora gw108 PASS\n', '27 ora gw109 PASS\n',
'27 ora gw110 PASS\n', '27 ora gw111 PASS\n', '27 ora gw112 PASS\n', '27 ora
gw113 PASS\n', '27 ora gw114 PASS\n', '27 ora gw115 PASS\n', '27 ora gw116
PASS\n', '27 ora gw117 PASS\n', '27 ora gw118 PASS\n', '27 ora gw120 PASS\n',
'27 ora gw121 PASS\n', '27 ora gw122 PASS\n', '27 ora gw123 PASS\n', '27 ora
gw124 PASS\n', '27 ora gw125 PASS\n', '27 ora gw150 PASS\n', '27 ora gw153
PASS\n', '27 ora gw155 FAIL\n', '27 ora gw156 PASS\n', '27 ora gw165 PASS\n',
'27 ora gw168 PASS\n', '27 ora gw169 PASS\n', '27 ora gw170 PASS\n', '27 ora
gw171 PASS\n', '27 ora gw172 PASS\n', '27 ora gw174 PASS\n', '27 ora gw175
PASS\n', '27 ora gw176 PASS\n', '27 ora gw178 PASS\n', '27 ora gw179 PASS\n',
'27 ora gw180 PASS\n', '27 ora gw181 PASS\n', '27 ora gw182 PASS\n', '27 ora
gw183 PASS\n', '27 ora gw184 PASS\n', '27 ora gw185 PASS\n', '27 ora gw186
PASS\n', '27 ora gw187 PASS\n', '27 ora gw188 PASS\n', '27 ora gw189 PASS\n',
'27 ora gw190 PASS\n', '27 ora gw191 PASS\n', '27 ora gw192 PASS\n', '27 ora
gw193 PASS\n', '27 ora gw194 PASS\n', '27 ora gw195 PASS\n', '27 ora gw196
FAIL\n', '27 ora gw200 PASS\n', '27 ora gw201 PASS\n', '27 ora gw202 PASS\n',
'27 ora gw203 PASS\n', '27 ora gw204 PASS\n', '27 ora gw205 PASS\n', '27 ora
gw206 PASS\n', '27 ora gw207 PASS\n', '27 ora gw209 PASS\n', '27 ora gw210
PASS\n', '27 ora gw231 PASS\n', '27 ora gw232 PASS\n', '27 ora gw260 PASS\n',
'27 ora gw320 PASS\n', '27 ora gw321 PASS\n', '27 ora gw322 PASS\n', '27 ora
gw400 PASS\n', '27 ora gw401 PASS\n', '27 ora gw402 FAIL\n', '27 ora gw403
PASS\n', '27 ora gw404 PASS\n', '27 ora gw405 PASS\n', '27 ora gw406 PASS\n',
'27 ora gw500 PASS\n', '27 ora gw501 PASS\n', '27 ora gw503 PASS\n', '27 ora
gw504 PASS\n', '27 ora gw505 PASS\n', '27 ora gw506 FAIL\n', '27 ora gw507
PASS\n', '27 ora gw508 PASS\n', '27 ora gw509 PASS\n', '27 ora gw510 PASS\n',
'27 ora gw511 PASS\n', '27 ora gw512 PASS\n', '27 ora gw513 PASS\n', '27 ora
gw514 PASS\n', '27 ora gw515 PASS\n', '27 ora gw516 PASS\n', '27 ora gw517
FAIL\n', '27 ora gw530 PASS\n', '27 ora gw532 PASS\n', '27 ora gw584 PASS\n',
'27 ora gw585 PASS\n', '27 ora gw586 PASS\n', '27 ora gw590 PASS\n', '27 ora
gw591 PASS\n', '27 ora gw592 PASS\n', '27 ora gw593 PASS\n', '27 ora gw594
PASS\n', '27 ora gw596 PASS\n', '27 ora gw597 PASS\n', '27 ora gw598 PASS\n',
'27 ora gw600 PASS\n', '27 ora gw601 PASS\n', '27 ora gw604 PASS\n', '27 ora
gw605 PASS\n', '27 ora gw606 PASS\n', '27 ora gw607 PASS\n', '27 ora gw608
PASS\n', '27 ora gw609 PASS\n', '27 ora gw611 PASS\n', '27 ora gw612 PASS\n',
'27 ora gw613 PASS\n', '27 ora gw614 PASS\n', '27 ora gw615 PASS\n', '27 ora
gw616 PASS\n', '27 ora gw650 PASS\n', '27 ora gw651 PASS\n', '27 ora gw653
PASS\n', '27 ora gw655 PASS\n', '27 ora gw656 PASS\n', '27 ora gw657 PASS\n',
'27 ora gw658 PASS\n', '27 ora gw659 FAIL\n', '27 ora gw662 PASS\n', '27 ora
gw663 FAIL\n', '27 ora gw680 PASS\n', '27 ora gw700 PASS\n', '27 ora gw701
PASS\n', '27 ora gw702 PASS\n', '27 ora gw703 PASS\n', '27 ora gw704 PASS\n',
'27 ora gw705 FAIL\n', '27 ora gw706 PASS\n', '27 ora gw707 PASS\n', '27 ora
gw708 PASS\n', '27 ora gw709 PASS\n', '27 ora gw710 PASS\n', '27 ora gw711
PASS\n', '27 ora gw713 PASS\n', '27 ora gw714 PASS\n', '27 ora gw715 PASS\n',
'27 ora gw718 PASS\n', '27 ora gw720 FAIL\n', '27 ora gw721 PASS\n', '27 ora
gw722 FAIL\n', '27 ora gw900 FAIL\n', '27 ora gw988 PASS\n', '27 ora gw989
PASS\n', '27 ora gw992 PASS\n', '27 ora gw993 PASS\n', '27 ora gw994 PASS\n',
'27 ora gw995 PASS\n', '27 ora gw996 PASS\n', '27 ora gw999 PASS\n']
print('difflib.Differ()')
d = difflib.Differ()
#diffs = list(d.compare(file1content, file2content))
print('d.compare(file1content, file2content)')
diffs = d.compare(file1content, file2content)
print(type(diffs))
print('for x in diffs: ## takes 2 mins!')
for x in diffs:
print 'x', x
#print('list(diffs)')
#diffs = list(diffs)
result = ''.join(diffs)
print(result)
--
Want to help the Review Board project? Donate today at
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~----------~----~----~----~------~----~------~--~---
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/reviewboard?hl=en
