We started with a local rescan that took:
121249081 function calls (121023286 primitive calls) in 665.469
CPU seconds
Latest profile info for Local Rescan shows:
17240440 function calls (17018317 primitive calls) in 506.979
CPU seconds
And the next bottleneck to tackle is cPickle.load, as per:
ncalls tottime percall cumtime percall filename:lineno(function)
354949 180.393 0.001 183.109 0.001 {cPickle.load}
2581 83.872 0.032 319.376 0.124
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:507(get_mdobjs_in_dir)
4539 42.603 0.009 42.603 0.009 {posix.listdir}
58545 36.384 0.001 36.384 0.001 {posix.remove}
58476 26.243 0.000 26.243 0.000 {method 'remove' of
'collections.deque' objects}
354949 16.301 0.000 16.301 0.000 {open}
37542 12.212 0.000 12.212 0.000 {posix.lstat}
459845 12.203 0.000 271.947 0.001
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:212(__getitem__)
886460 10.724 0.000 17.639 0.000
/usr/lib/python2.6/posixpath.py:59(join)
354949 9.861 0.000 211.277 0.001
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:118(_load_pickle)
459845 8.220 0.000 11.197 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:292(purge)
393979 6.764 0.000 23.727 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:67(key_file)
459845 4.862 0.000 7.537 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:256(__getitem__)
2375292 3.857 0.000 3.857 0.000 {len}
1732359 3.702 0.000 3.702 0.000 {method 'startswith' of 'str'
objects}
459845 3.496 0.000 5.244 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:287(update)
218207 3.465 0.000 46.556 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:950(changed)
444246/222123 3.455 0.000 4.842 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:141(__init__)
1674418 3.345 0.000 3.345 0.000 {method 'endswith' of 'str'
objects}
354949 3.238 0.000 235.622 0.001
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:142(__getitem__)
354949 2.819 0.000 16.586 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:270(__setitem__)
2581 2.778 0.001 369.058 0.143
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/local_rescan.py:182(_get_share_info)
354813 2.715 0.000 2.715 0.000
/usr/lib/python2.6/os.py:721(_make_stat_result)
218207 1.992 0.000 3.211 0.000
/usr/lib/python2.6/posixpath.py:79(split)
900176 1.743 0.000 1.743 0.000 {method 'get' of 'dict' objects}
354949 1.474 0.000 184.583 0.001
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:111(_unpickle)
2581 1.443 0.001 77.223 0.030
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/local_rescan.py:494(_paths_filter)
444246 1.387 0.000 1.387 0.000 {method 'update' of 'dict'
objects}
715986 1.305 0.000 1.305 0.000 {method 'append' of 'list'
objects}
218207 1.268 0.000 1.943 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:922(_get_mdid_from_args)
2581 1.160 0.000 506.973 0.196
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/local_rescan.py:545(inner_scan)
81745 1.114 0.000 1.114 0.000
/usr/lib/python2.6/logging/__init__.py:1222(getEffectiveLevel)
19515 1.036 0.000 1.872 0.000
/usr/lib/python2.6/posixpath.py:308(normpath)
19515 0.856 0.000 39.860 0.002
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:163(__delitem__)
474845 0.817 0.000 0.817 0.000 {method 'append' of
'collections.deque' objects}
2581 0.816 0.000 450.035 0.174
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/local_rescan.py:231(_compare)
458022 0.763 0.000 0.763 0.000 {method 'popleft' of
'collections.deque' objects}
19515 0.700 0.000 71.420 0.004
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:678(delete_metadata)
81745 0.633 0.000 2.214 0.000
/usr/lib/python2.6/logging/__init__.py:1096(log)
19515 0.618 0.000 67.340 0.003
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:228(__delitem__)
354949 0.532 0.000 0.532 0.000 {method '__enter__' of 'file'
objects}
19515 0.519 0.000 26.808 0.001
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:276(__delitem__)
222126 0.508 0.000 0.508 0.000 {method 'rfind' of 'str' objects}
81745 0.467 0.000 1.581 0.000
/usr/lib/python2.6/logging/__init__.py:1236(isEnabledFor)
222003 0.418 0.000 0.418 0.000 {method 'rstrip' of 'str' objects}
37530 0.403 0.000 0.403 0.000 {posix.access}
110027 0.177 0.000 0.177 0.000
/usr/lib/python2.6/stat.py:24(S_IFMT)
23431 0.167 0.000 0.167 0.000 {method 'split' of 'str' objects}
37542 0.147 0.000 12.359 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:1155(get_stat)
37542 0.137 0.000 0.200 0.000
/usr/lib/python2.6/stat.py:55(S_ISLNK)
37530 0.130 0.000 0.189 0.000
/usr/lib/python2.6/stat.py:40(S_ISDIR)
34955 0.119 0.000 0.174 0.000
/usr/lib/python2.6/stat.py:49(S_ISREG)
19515 0.101 0.000 0.101 0.000 {method 'join' of 'str' objects}
19516 0.096 0.000 0.096 0.000 {isinstance}
3916 0.088 0.000 0.381 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:543(get_by_mdid)
37510 0.075 0.000 0.075 0.000 {method 'pop' of 'dict' objects}
3659 0.058 0.000 0.058 0.000 {posix.stat}
19515 0.055 0.000 0.055 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/file_shelf.py:283(__contains__)
3916 0.048 0.000 0.075 0.000
/usr/lib/python2.6/posixpath.py:117(dirname)
3916 0.028 0.000 0.057 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:1048(get_abspath)
3659 0.025 0.000 0.083 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/local_rescan.py:265(check_stat)
3916 0.021 0.000 0.029 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/filesystem_manager.py:1039(_get_share)
5162 0.012 0.000 0.012 0.000 {method 'iteritems' of 'dict'
objects}
2581 0.006 0.000 0.006 0.000 {method 'extend' of 'list'
objects}
2581 0.005 0.000 0.005 0.000 {method 'disable' of
'_lsprof.Profiler' objects}
10 0.001 0.000 0.001 0.000 {sum}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:226(__init__)
1 0.000 0.000 0.000 0.000 {method 'flush' of 'file' objects}
2 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:382(formatTime)
1 0.000 0.000 0.000 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/logger.py:86(shouldRollover)
2 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:423(format)
10 0.000 0.000 0.000 0.000 {method 'itervalues' of 'dict'
objects}
2 0.000 0.000 0.000 0.000 {time.localtime}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:756(emit)
2 0.000 0.000 0.000 0.000 {time.strftime}
2 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:637(format)
2 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:289(getMessage)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:1113(findCaller)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:1196(callHandlers)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:660(handle)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/threading.py:116(acquire)
1 0.000 0.000 0.001 0.001
/usr/lib/python2.6/logging/__init__.py:1147(_log)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/handlers.py:63(emit)
3 0.000 0.000 0.000 0.000
/usr/lib/python2.6/string.py:351(find)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/threading.py:136(release)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:148(getLevelName)
6 0.000 0.000 0.000 0.000 {hasattr}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/genericpath.py:85(_splitext)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:1167(handle)
2 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:560(filter)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:520(filter)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:1134(makeRecord)
1 0.000 0.000 0.000 0.000 {method 'seek' of 'file' objects}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:76(<lambda>)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:749(flush)
2 0.000 0.000 0.000 0.000 {time.time}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:843(emit)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/handlers.py:269(shouldRollover)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/posixpath.py:109(basename)
3 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
1 0.000 0.000 0.000 0.000
/home/rtg/Projects/ubuntuone/speed-up-that-local-rescan/ubuntuone/syncdaemon/logger.py:124(filter)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:617(acquire)
4 0.000 0.000 0.000 0.000 {thread.get_ident}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/threading.py:808(currentThread)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/logging/__init__.py:624(release)
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/posixpath.py:94(splitext)
1 0.000 0.000 0.000 0.000 {method 'write' of 'file' objects}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/threading.py:662(name)
2 0.000 0.000 0.000 0.000
/usr/lib/python2.6/threading.py:64(_note)
1 0.000 0.000 0.000 0.000 {method 'tell' of 'file' objects}
1 0.000 0.000 0.000 0.000 {built-in method acquire}
1 0.000 0.000 0.000 0.000 {sys._getframe}
1 0.000 0.000 0.000 0.000 {built-in method release}
1 0.000 0.000 0.000 0.000 {posix.getpid}
1 0.000 0.000 0.000 0.000
/usr/lib/python2.6/posixpath.py:42(normcase)
--
LR should be faster when getting file stats
https://bugs.launchpad.net/bugs/476711
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
