On Jul 1, 2009, at 11:16 AM, Greg Klanderman wrote: >>>>>> andyg <[email protected]> >>>>>> writes: > >> I've changed all scanning in the current 7.4 over to the new scanner >> code I've been working on. I expect there will be bugs. I've >> started a >> wiki page that attempts to describe how the new scanner in 7.4 works, >> please read it and let me know if you need more info or have >> problems. > >> http://wiki.slimdevices.com/index.php/NewScanner > > This sounds like great stuff, Andy, I'm excited to check it out! > > In automatic rescan mode, once it detects a song has changed, how long > does it take to update the database? Will it still be doing full > versions of all the slow post-scan steps, or will those be made > incremental and fast?
The rescan code fires 15 seconds after the last change. If using a network share, the delay is a bit longer. > How about rescanning playlists? This is probably my biggest > frustration right now - I edit a playlist then to get the changes seen > by the server I need to manually rescan, and because of the post-scan > steps that takes a significant amount of time. It should at least be > smart enough to skip the post-scan steps if no tag changes are found > rescanning the playlists, which should usually be the case unless your > playlist changes refer to (new) files outside the server's normal > music directory. This could be a big win for very little effort. Yeah playlists are also rescanned this way (although not well tested yet). > On that web page you write: > > | Scanning of file metadata and tags accounted for approx. 1/3 of the > | scan time in prior versions of SqueezeCenter. All the Perl-based > file > | scanning modules have been replaced by a single C-based module, > | Audio::Scan. This reduces this metadata/tag reading time to a much > | smaller fraction of the scan process. The rest of the scan time is > | taken in the Perl database abstraction layer (DBIx::Class), and > | remains generally unchanged. > > By what fraction has the tag scanning been sped up? > > Attacking the component taking 1/3 of the time is certainly useful, > but usually one would first go after the component taking 2/3.. any > plans to speed up the database access? From what I understand DBIx is > really slow, is there a plan to use something else? Also, I would > think it'd be a big win to cache updates to the database and only > commit in bigger chunks while scanning. Of course, but improving the performance of DBIx::Class is not a trivial task. Definitely something we need to improve, no question. > I've got about 22,000 mp3s, and can extract the tags in less than 45 > seconds so I really do not see why a full rescan should take any more > than about a minute: > > [mp3] g...@lwm| find . -name '*.mp3' -print | wc -l > 22162 > [mp3] g...@lwm| time find . -name '*.mp3' -print0 | xargs -0 id3info > | wc -c > 11067476 > find . -name '*.mp3' -print0 0.03s user 0.07s system 0% cpu 41.251 > total > xargs -0 id3info 12.63s user 28.98s system 96% cpu 43.314 total > wc -c 0.12s user 0.96s system 2% cpu 43.311 total > > And that's only about 10 megs of tag data (probably a lot less if you > consider how verbose the output of id3info is); there's no reason I > see not to store all the tag data in ram while scanning and commit to > the DB all at once at the end. I don't really like comments like this; you can comparing apples and oranges. You are completely ignoring the large amount of work SC does once it gets your tag information. Raw tag scanning is fast, and Audio::Scan is very fast by itself. But then you have to do something with the tags, translate them into the right values, put them in the database, etc. Sure, you could write all of that stuff in C but matching the features in SC would be an incredible amount of work. Netgear tried this and ended up with a really fast scanner with maybe 10% of the features. Porting the tag scanning code to C was a manageable amount of work and made sense to live as a standalone module. That's why I did it. _______________________________________________ beta mailing list [email protected] http://lists.slimdevices.com/mailman/listinfo/beta
