Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?
On Fri, Jun 12, 2009 at 04:33:43PM +0200, Richard van den Berg wrote: On Fri, June 12, 2009 15:21, Heinrich Langos wrote: I'd like to have a chain like this: ok - ok - broken Here is such a chain: http://richard.vdberg.org/gnupod/ok/11181/iTunesDB http://richard.vdberg.org/gnupod/ok/11182/iTunesDB http://richard.vdberg.org/gnupod/ok/11183/iTunesDB http://richard.vdberg.org/gnupod/not_ok/11184/iTunesDB Does this file have anything in its attributes that the files that you added successfully (until #11531) did not have? artwork maybe? Nope. In fact it breaks in the middle of an album. All those songs had been added with the same gnupod_addsong.pl --artwork run. I took a look at that chain (11182-11183-11184) using hexdump, sed and diff and i came pretty much the the same conclusion... Nothing obviously wrong. I wonder if it has anything to do with the IDs that playlist entries get. currently they are just sequentially continued after the last track ID. Thats why you get such a large number of tiny differences in the master playlist even when adding just one track. I don't have much time this weekend but maybe you could check out what happens if you start giving those IDs from the highest value and decrement with each entry. I doubt that this will solve the problem but it probably will help decrease the noise in the diffs. BTW thanks for the hint with vbindiff. I was using some mixture of hexdump, sed (to do linebreaks on mh.. patterns) and diff. Could you try to add the SAME file over and over until you get a non working state? That would make hunt for an overflow in the internal structures even easier. (gnupod_addsong -d allow duplicates). I can do so tonight, but I was rather hoping of finding this bug without having to wipe my iPod. Can't I just create a fake GNUtunesDB.xml with the same file over and over again? I don't have the space left to add another 11000 actual songs to my iPod (unless it's 1 kb or smaller). In my testing so far I've just been manually editing GNUtunesDB.xml and running mktunes.pl to generate a new iTunesDB. Skip that. I was hoping to find some pattern in the added file's characteristic but that will not help. Best way to get there is a binary search. That's exactly what I used last night. :-) 15 iterations for 3 files is the best you can do. Good to kown that we can talk more than just basic patterns. Did you try adding your collection with gtkpod? I assume it will be much faster as it is not written in perl. Nope. I really don't need a GUI and yet another local database to keep track off. I want gnupod! :-) Me too, I was just wondering if their code produced an iTunesDB that wouldn't have the same problem. Then we could look for the differences. cheers -henrik ___ Bug-gnupod mailing list Bug-gnupod@nongnu.org http://lists.nongnu.org/mailman/listinfo/bug-gnupod
[Bug-gnupod] Performance of gnupod on large data sets
On Fri, Jun 12, 2009 at 09:51:18PM +0200, Richard van den Berg wrote: To get some numbers on the performance you could do me a favor and add a single file to the rather full ipod while profiling the perl process. The results are at: With 11600 files http://richard.vdberg.org/gnupod/dprof/run1 With 30415 files http://richard.vdberg.org/gnupod/dprof/run3 After the first run please move the tmon.out file aside and run the same perl -d:DProf /path/to/gnupod_addsong.pl /path/to/file.mp3 line once more. (This time gnupod_addsong should abort before writing the file to your ipod because of the duplicate detection.) The results are at: With 11600 files http://richard.vdberg.org/gnupod/dprof/run2 With 30415 files http://richard.vdberg.org/gnupod/dprof/run4 I did another run with the --artwork switch at http://richard.vdberg.org/gnupod/dprof/run5 I took a glance at the stats and I didn't know that artwork does slow down the process that much. About xescaped.. I'll add a simple check at the start (after taking care of the first couple of XML entities) that will return the string if there are no characters outside the plain ascii range 0x20-0x7F.. thus skipping any of the later regex matches and conversions that only apply to utf8 stuff. cheers -henrik ___ Bug-gnupod mailing list Bug-gnupod@nongnu.org http://lists.nongnu.org/mailman/listinfo/bug-gnupod
Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?
The results I got today were not consistent with yesterday (ok iTunesDBs
would now be not ok). I learned a lot about the binary format, and
figured out that by messing with the ArtworkDB (for the performance
tests) the dbid_1 numbers were now out of whack. However, restoring the
whole iPod_Control/Artwork directory from yesterday didn't help.
So I decided to clear the iPod_Control/Artwork directory and take out
the dbid_1, artworkcnt and has_artwork parameters from my GNUtunesDB.xml
files. I had to do a complete new binary search to find the following ok
- ok - not_ok sequence:
http://richard.vdberg.org/gnupod/no_artwork/ok/11124/iTunesDB
http://richard.vdberg.org/gnupod/no_artwork/ok/11125/iTunesDB
http://richard.vdberg.org/gnupod/no_artwork/not_ok/11126/iTunesDB
This time file number 11126 seems to be the problem:
file addtime=3327445665 album=Love Train - The Sound Of
Philadelphia - Disc 2 artist=Billy Paul bitrate=215 cdnum=0
cds=0 comment= compilation=1 composer=
desc=/d01/mp3/marcel/Love Train/The Sound Of Philadelphia/32 - Billy
Paul - Thanks For Saving My Life.mp3 fdesc=MPEG 1 layer 3 file
filesize=4813000 genre=Soul id=11152 mediatype=1
path=:iPod_Control:Music:f23:g0_For_Saving_My_Life.mp3 playcount=0
songnum=12 songs=0 soundcheck=5070 srate=44100 time=178311
title=Thanks For Saving My Life volume=0 year=2008 /
On 6/13/09 12:43 PM, H. Langos wrote:
I don't have much time this weekend but maybe you could check out what
happens if you start giving those IDs from the highest value and decrement
with each entry.
Interesting idea. See attached patch. It does make the diff a whole lot
easier to read. And it made a difference as well. With this patch
applied 11126 suddenly worked.
During the binary search for the next limit I found that if an iTunesDB
does not work right after I unplug the USB, it *might* work after a
reset of the iPod. How weird is that? I verified that after the reboot
the iTunesDB is exactly (cmp -bl) the same as before.
I'm now starting to think it's a RAM issue. According to
http://ipodlinux.org/wiki/Generations#Fifth_Generation_.285G.29_.2F_Fifth_Generation_Enhanced_.285.5G.29
my 30GB turned 240GB iPod has 32MB of RAM. Maybe I should try to get a
hold of a 60GB/80GB version (64GB of RAM)?
Using a binary search including a reset cycle, I was able to get it to
work with 17759 files (then I got bored):
25026442 not_ok/18053/iTunesDB-plid
25826240 not_ok/18640/iTunesDB-plid
28968870 not_ok/20987/iTunesDB-plid
42047276 not_ok/30415/iTunesDB-plid
22566800 ok/16293/iTunesDB-plid
24192148 ok/17466/iTunesDB-plid
24621122 ok/17759/iTunesDB-plid
You can find the files here: http://richard.vdberg.org/gnupod/no_artwork/
I used the plid extension to show I've used the decrement-plid patch
to produce those iTunesDBs.
Cheers,
Richard
? .gnupod_version
? Makefile
? autom4te.cache
? config.log
? config.status
? configure
Index: src/ext/Mktunes.pm
===
RCS file: /sources/gnupod/gnupod/src/ext/Mktunes.pm,v
retrieving revision 1.6
diff -u -r1.6 Mktunes.pm
--- src/ext/Mktunes.pm 6 Oct 2007 07:26:52 - 1.6
+++ src/ext/Mktunes.pm 13 Jun 2009 17:05:41 -
@@ -12,7 +12,7 @@
my($class,%args) = @_;
my $self = { Connection=$args{Connection}, Mode=MODE_ADDFILE,
Artwork=$args{Artwork},
-ArrayFiles = [], CountFiles = 0, Sequence = 0,
iPodName = $args{iPodName},
+ArrayFiles = [], CountFiles = 0, Sequence = 0,
PlSequence = 0x, iPodName = $args{iPodName},
MasterPlaylist = [], Playlists = {},
SmartPlaylists = {},
FuzzyDb_Normal = {}, FuzzyDb_Lowercase = {} };
bless($self,$class);
@@ -91,6 +91,8 @@
sub GetFileCount{ my($self) = @_; return $self-{CountFiles};
}
# Increments Sequence counter
sub GetNextId { my($self) = @_; return ++$self-{Sequence};
}
+ # Decrements PlSequence counter
+ sub GetPrevId { my($self) = @_; return --$self-{PlSequence};
}
# Dispatch connector
sub GetConnection { my($self) = @_; return $self-{Connection}
}
# Returns array to files
@@ -240,7 +242,7 @@
foreach my $fqid (@{$cont}) {
- my $current_id = $self-GetNextId;
+ my $current_id = $self-GetPrevId;
my $current_mhod =
GNUpod::iTunesDB::mk_mhod({fqid=$fqid});
my $current_mhip =
GNUpod::iTunesDB::mk_mhip({childs = 1, plid = $current_id, sid=$fqid,
size=length($current_mhod)});
next unless (defined($current_mhod)
defined($current_mhip));
___
