Re: Mac: kbuildsycoca5 and kded5 deadloop *somewhere* after upgrading to Qt 5.7.1

[René J . V . Bertin]

On Saturday December 31 2016 14:10:38 David Faure wrote:

>OK that recursion is fixed in https://git.reviewboard.kde.org/r/129733/
>
>Can you check if you still hit an issue?

Sure ... next year ;)
I do wonder, won't there be unwanted/unforeseen side-effects from skipping 
symlinks to directories? 

>I'm wondering if the other recursion (the one in 
>VFolderMenu::loadApplications) hits you as well, the fix is only for the 
>recursive timestamp checking, not for the code that actually does the parsing.

I guess we'll see when I recreate that symlink but the chances seem slim if it 
didn't already show up in the backtrace I uploaded.
>


> if (fi.isDir() && !fi.isBundle()) {
>
>which makes sense if we don't want to go into bundles to look for application 
>desktop files. So we might want to do in sycocautils_p.h as well.

Indeed. But what about app bundle builds which may have `isBundle() == true` 
for their "root"? Or are they supposed not to be using .desktop files and 
whatever else sycoca trawls?

R.

Re: Mac: kbuildsycoca5 and kded5 deadloop *somewhere* after upgrading to Qt 5.7.1

[David Faure]

On vendredi 30 décembre 2016 19:29:35 CET René J.V. Bertin wrote:
>   frame #17: 0x000106a6f89c libKF5Service.5.dylib`bool
> KSycocaUtilsPrivate::visitResourceDirectoryHelper me=, visitor=)::$_4>(QString const&,
> KBuildSycoca::build()::$_4) + 348 at ksycocautils_p.h:47 

OK that recursion is fixed in https://git.reviewboard.kde.org/r/129733/

Can you check if you still hit an issue?
I'm wondering if the other recursion (the one in 
VFolderMenu::loadApplications) hits you as well, the fix is only for the 
recursive timestamp checking, not for the code that actually does the parsing.

Actually, you probably won't hit the issue there because the code in 
vfoldermenu says:

 if (fi.isDir() && !fi.isBundle()) {

which makes sense if we don't want to go into bundles to look for application 
desktop files. So we might want to do in sycocautils_p.h as well.

Maybe both should say

 if (fi.isDir() && !fi.isSymLink() && !fi.isBundle()) {


-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

Re: Mac: kbuildsycoca5 and kded5 deadloop *somewhere* after upgrading to Qt 5.7.1

[René J . V . Bertin]

For the sake of completeness, here are the QSP locations returned (with 
"XDG-compliant" writable locations), for 5.6.2 and 5.7.1 .

R.  DesktopLocation: "Desktop" = */Users/bertin/Desktop*
  DocumentsLocation: "Documents" = */Users/bertin/Documents*
  FontsLocation: "Fonts" = */Users/bertin/Library/Fonts* /Library/Fonts 
/System/Library/Fonts
  ApplicationsLocation: "Applications" = 
*/Users/bertin/.local/share/applications* /opt/local/share/applications
  MusicLocation: "Music" = */Users/bertin/Music*
  MoviesLocation: "Movies" = */Users/bertin/Movies*
  PicturesLocation: "Pictures" = */Users/bertin/Pictures*
  TempLocation: "TemporaryItems" = 
*/var/folders/j1/1439ppj08xj8h6006s6drbq0gs/T*
  HomeLocation: "Home" = */Users/bertin*
  DataLocation: "Application Support" = 
*/Users/bertin/.local/share/QtProject/qtdiag* /Library/Application 
Support/QtProject/qtdiag /opt/local/share/QtProject/qtdiag 
/opt/local/libexec/qt5/bin/
  AppLocalDataLocation: "Application Support" = 
*/Users/bertin/.local/share/QtProject/qtdiag* /Library/Application 
Support/QtProject/qtdiag /opt/local/share/QtProject/qtdiag 
/opt/local/libexec/qt5/bin/
  CacheLocation: "Caches" = */Users/bertin/.cache/QtProject/qtdiag* 
/Users/bertin/.cache /Library/Caches/QtProject/qtdiag
  GenericDataLocation: "Application Support" = */Users/bertin/.local/share* 
/opt/local/share /Library/Application Support
  RuntimeLocation: "Application Support" = 
*/var/folders/j1/1439ppj08xj8h6006s6drbq0gs/T/runtime-bertin*
  ConfigLocation: "Preferences" = */Users/bertin/.config* /opt/local/etc/xdg
  DownloadLocation: "Downloads" = */Users/bertin/Downloads*
  GenericCacheLocation: "Caches" = */Users/bertin/.cache* /Users/bertin/.cache 
/Library/Caches
  GenericConfigLocation: "Preferences" = */Users/bertin/.config* 
/opt/local/etc/xdg
  AppDataLocation: "Application Support" = 
*/Users/bertin/.local/share/QtProject/qtdiag* /Library/Application 
Support/QtProject/qtdiag /opt/local/share/QtProject/qtdiag 
/opt/local/libexec/qt5/bin/
  AppConfigLocation: "Preferences" = 
*/Users/bertin/Library/Preferences/QtProject/qtdiag*
  DesktopLocation: "Desktop" = */Users/bertin/Desktop*
  DocumentsLocation: "Documents" = */Users/bertin/Documents*
  FontsLocation: "Fonts" = */Users/bertin/Library/Fonts* /Library/Fonts 
/System/Library/Fonts
  ApplicationsLocation: "Applications" = 
*/Users/bertin/.local/share/applications* /Users/bertin/Applications 
/opt/local/share/applications /Applications
  MusicLocation: "Music" = */Users/bertin/Music*
  MoviesLocation: "Movies" = */Users/bertin/Movies*
  PicturesLocation: "Pictures" = */Users/bertin/Pictures*
  TempLocation: "Temporary Items" = 
*/var/folders/j1/1439ppj08xj8h6006s6drbq0gs/T*
  HomeLocation: "Home" = */Users/bertin*
  DataLocation: "Application Support" = 
*/Users/bertin/.local/share/QtProject/qtdiag* /Users/bertin/Library/Application 
Support/QtProject/qtdiag /Library/Application Support/QtProject/qtdiag 
/opt/local/share/QtProject/qtdiag /opt/local/libexec/qt5/bin/
  AppLocalDataLocation: "Application Support" = 
*/Users/bertin/.local/share/QtProject/qtdiag* /Users/bertin/Library/Application 
Support/QtProject/qtdiag /Library/Application Support/QtProject/qtdiag 
/opt/local/share/QtProject/qtdiag /opt/local/libexec/qt5/bin/
  CacheLocation: "Caches" = */Users/bertin/.cache/QtProject/qtdiag* 
/Users/bertin/Library/Caches/QtProject/qtdiag /Users/bertin/.cache 
/Library/Caches/QtProject/qtdiag
  GenericDataLocation: "share" = */Users/bertin/.local/share* 
/Users/bertin/Library/Application Support /opt/local/share /Library/Application 
Support
  RuntimeLocation: "runtime-bertin" = 
*/var/folders/j1/1439ppj08xj8h6006s6drbq0gs/T/runtime-bertin* 
/Users/bertin/Library/Application Support
  ConfigLocation: ".config" = */Users/bertin/.config* 
/Users/bertin/Library/Preferences /opt/local/etc/xdg
  DownloadLocation: "Downloads" = */Users/bertin/Downloads*
  GenericCacheLocation: ".cache" = */Users/bertin/.cache* 
/Users/bertin/Library/Caches /Users/bertin/.cache /Library/Caches 
/System/Library/Caches
  GenericConfigLocation: ".config" = */Users/bertin/.config* 
/Users/bertin/Library/Preferences /opt/local/etc/xdg
  AppDataLocation: "Application Support" = 
*/Users/bertin/.local/share/QtProject/qtdiag* /Users/bertin/Library/Application 
Support/QtProject/qtdiag /Library/Application Support/QtProject/qtdiag 
/opt/local/share/QtProject/qtdiag /opt/local/libexec/qt5/bin/
  AppConfigLocation: "Preferences" = 
*/Users/bertin/Library/Preferences/QtProject/qtdiag*

Re: Mac: kbuildsycoca5 and kded5 deadloop *somewhere* after upgrading to Qt 5.7.1

[René J . V . Bertin]

On Friday December 30 2016 18:49:46 David Faure wrote:

> > Edit: it doesn't.
> 
> What's it and what is it that it doesn't?

Sorry, yes.

> Can you reproduce the problem on Linux or not?

No. kbuildsycoca5 finishes without issues there. It may run a bit slower but 
that could just as well be my imagination.

> Anyhow, please post backtraces. Anything else is just guessing.

OK, attaching lldb after letting kbuildsycoca5 --noincremental run for about 
15min. while I made a fresh cuppa:

* thread #1: tid = 0x21d12b0, 0x7fff8ca0c0aa 
libsystem_kernel.dylib`__getattrlist + 10, queue = 'com.apple.main-thread', 
stop reason = signal SIGSTOP
  * frame #0: 0x7fff8ca0c0aa libsystem_kernel.dylib`__getattrlist + 10
frame #1: 0x7fff8f949da7 
CoreServicesInternal`corePropertyProviderPrepareValues(__CFURL const*, 
__FileCache*, __CFString const* const*, void const**, long, void const*, 
__CFError**) + 484
frame #2: 0x7fff8f93ffbb 
CoreServicesInternal`prepareValuesForBitmap(__CFURL const*, __FileCache*, 
_FilePropertyBitmap*, __CFError**) + 227
frame #3: 0x7fff8f93dca7 
CoreServicesInternal`_FSURLCopyResourcePropertyForKeyInternal(__CFURL const*, 
__CFString const*, void*, void*, __CFError**, unsigned char) + 213
frame #4: 0x7fff8f93dbcd 
CoreServicesInternal`_FSURLCopyResourcePropertyForKey + 15
frame #5: 0x7fff866a02bb CoreFoundation`CFURLCopyResourcePropertyForKey 
+ 123
frame #6: 0x000106e5bf5c 
QtCore`hasResourcePropertyFlag(data=, entry=, 
key=) + 172 at qfilesystemengine_unix.cpp:86
frame #7: 0x000106e5a549 
QtCore`QFileSystemEngine::fillMetaData(entry=0x7f80a96ae078, 
data=0x7f80a96ae090, what=) + 649 at 
qfilesystemengine_unix.cpp:529
frame #8: 0x000106dff291 QtCore`QFileInfo::isHidden(this=) 
const + 81 at qfileinfo.cpp:970
frame #9: 0x000106df7343 
QtCore`QDirIteratorPrivate::matchesFilters(this=0x7f80a96ade70, 
fileName=, fi=0x7fff591e4ae0) const + 387 at 
qdiriterator.cpp:358
frame #10: 0x000106df6c45 
QtCore`QDirIteratorPrivate::advance(this=0x7f80a96ade70) + 1045 at 
qdiriterator.cpp:255
frame #11: 0x000106df61c6 
QtCore`QDirIteratorPrivate::QDirIteratorPrivate(this=, 
entry=, nameFilters=, resolveEngine=, 
filters=, flags=) + 1206 at qdiriterator.cpp:175
frame #12: 0x000106df7aa6 
QtCore`QDirIterator::QDirIterator(this=0x7fff591e4d28, path=, 
nameFilters=, filters=, flags=) + 86 at 
qdiriterator.cpp:465
frame #13: 0x000106ded253 
QtCore`QDir::entryInfoList(this=, nameFilters=, 
filters=, sort=) const + 275 at qdir.cpp:1371
frame #14: 0x000106ded12c 
QtCore`QDir::entryInfoList(this=, filters=, 
sort=) const + 28 at qdir.cpp:1294
frame #15: 0x000106a6f77a libKF5Service.5.dylib`bool 
KSycocaUtilsPrivate::visitResourceDirectoryHelper(QString const&, KBuildSycoca::build()::$_4) + 58 
at ksycocautils_p.h:39
frame #16: 0x000106a6f89c libKF5Service.5.dylib`bool 
KSycocaUtilsPrivate::visitResourceDirectoryHelper(QString const&, KBuildSycoca::build()::$_4) + 348 
at ksycocautils_p.h:47
frame #17: 0x000106a6f89c libKF5Service.5.dylib`bool 
KSycocaUtilsPrivate::visitResourceDirectoryHelper(QString const&, KBuildSycoca::build()::$_4) + 348 
at ksycocautils_p.h:47


https://paste.kde.org/ptnxccplk/upymg1/raw

Frame #7 gives me the file being handled: 
/Applications/Multimedia/Video/iDVD.app/Contents/Frameworks/DVDBase.framework/Versions/A/Frameworks/DVDBase.framework/Versions/A/Frameworks/DVDBase.framework/Versions/A/Frameworks/DVDBase.framework/Versions/A/Frameworks/DVDBase.framework/Versions/A/Framew...

Indeed, 
"/Applications/Multimedia/Video/iDVD.app/Contents/Frameworks/DVDBase.framework/Versions/A/Frameworks"
 was a symlink to "../../../". Not a circular link, but something like it (is 
there a name for this kind of situation?).

After removing that link the process terminated, eventually.

A subsequent regular run took "only" 2:13 minutes at 40% CPU. A few well-placed 
qDebug() statements confirm that most of that time is spent trawling through 
/Applications (including all of Xcode's 5+Gb).

I think we'll have to consider skipping known Mac locations that are highly 
unlikely to contain anything sycoca relevant. Or maybe any generic locations 
that aren't user-specific and are not under the install prefix. At least as a 
MacPorts-specific patch.
Do you have suggestions where and how to filter out at least /Applications?

R.

Re: Mac: kbuildsycoca5 and kded5 deadloop *somewhere* after upgrading to Qt 5.7.1

[David Faure]

On vendredi 30 décembre 2016 18:19:17 CET René J.V. Bertin wrote:
> On Friday December 30 2016 18:08:11 René J.V. Bertin wrote:
> > My analoguous Linux build isn't finished yet so I cannot say whether this
> > is Mac-specific or something that may have to do with the way I install
> > things.
>
> Edit: it doesn't.

What's it and what is it that it doesn't?
Too many possible subjects in the previous sentence.
Can you reproduce the problem on Linux or not?
 
Anyhow, please post backtraces. Anything else is just guessing.

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

Re: Mac: kbuildsycoca5 and kded5 deadloop *somewhere* after upgrading to Qt 5.7.1

[René J . V . Bertin]

On Friday December 30 2016 18:08:11 René J.V. Bertin wrote:

> My analoguous Linux build isn't finished yet so I cannot say whether this is 
> Mac-specific or something that may have to do with the way I install things.

Edit: it doesn't. 

And FWIW, I did rebuild KService and KDEd against the new Qt version.

R.

Mac: kbuildsycoca5 and kded5 deadloop *somewhere* after upgrading to Qt 5.7.1

[René J . V . Bertin]

Hi,

I am in the process of upgrading to Qt 5.7.1 . While porting my Qt5 patches and 
building was largely a no-brainer and most applications seem to work without 
issues, there are (at least) 2 annoying exceptions among basic KF5 apps:

- kded5 deadloops somewhere
- kbuildsysoca5 idem.

In addition, my KDevelop 4.7 install that works fine when I have Qt 5.6.2 
active now crashes randomly because of an illegal operation *in a Mac SDK 
library*. That doesn't seem to make sense, yet it's reproducible.

On Mac we might be able to do without kded5, but I don't think we can do 
without kbuildsycoca5 .
My analoguous Linux build isn't finished yet so I cannot say whether this is 
Mac-specific or something that may have to do with the way I install things.

Does anyone have any suggestion where I might start looking? I already tried to 
run under lldb, interrupt once the deadloop was clearly installed and try to 
step out of functions to get an idea of that loop. Didn't really work; at some 
point lldb just stops co-operating ("error: Couldn't find thread plan to 
implement step type.").

There is a difference between 5.6.2 and 5.7.1 in the native QStandardPaths 
locations that are returned. I don't use those for writing, but if for instance 
kbuildsysoca5 parses everything under the GenericCacheLocation it will now have 
to wade through over 3.5Gb of crud, and that would of course take "a bit" more 
time.

Thanks...

R.