Sorry I've been busy with other things for a while.
Thanks for tracking this down, and posting a proposed fix (on another
thread).
Can you email me (directly, not via the list) one of the V3 attrib files
that fails with V4? I'd like to be 100% sure I understand the bug and fix.
Thanks,
Craig
On Sat, Apr 6, 2019 at 9:07 PM <backu...@kosowsky.org> wrote:
> backu...@kosowsky.org wrote at about 23:48:28 -0400 on Saturday, April 6,
> 2019:
> > backu...@kosowsky.org wrote at about 23:12:32 -0400 on Saturday, April
> 6, 2019:
> > > backu...@kosowsky.org wrote at about 14:58:10 -0400 on Sunday,
> March 31, 2019:
> > > >
> > > > When I run 'BackupPC_migrateV3toV4' (v4.3.0), I get occasional
> errors
> > > > of form:
> > > > BackupPC_migrateV3toV4: can't read attribute file
> ./f%2fdata/fmedia/f0/fAndroid/fdata/fcom.ebay.redlaser/fcache/fhttp/attrib
> > > >
> > > > Plus an associated error:
> > > > bpc_attrib_dirRead: got unreasonable file name length 852354
> > > >
> > > > Similarly if I try to read the attrib file directly using
> BackupPC_attribPrint, I get:
> > > > BackupPC_attribPrint: cannot read attrib file
> <path-to-file>/f%2fdata/fmedia/f0/fAndroid/fdata/fcom.ebay.redlaser/fcache/fhttp/attrib
> > > >
> > > > HOWEVER, if I use the v3.1 version of BackupPC_attribPrint, the
> attrib file prints out just fine!!!
> > > > SO, the issue seems to be with how v4.x reads attrib files
> > > >
> > > > And none of the filenames are longer than 255 characters...
> > > > For example the longest is:
> > > > fcache_http%253A%252F%252Fwww.swap.com
> %252Fi%252FM%252FAMIfv95PqwhrAhpZvHMI9wTeAMmp74SepSA1FcgE-HwOBvARxlhxJ2MNyBqHSNkxbLWiKa50yAM6dEAP7CA_DpByvhW4MB4wp6KPcmAzKpcUSgAliHCnZ4OTQ8NbA3xOUfehqKnpdFXq-xHdSJ3aU09RXP-0pfnFVDH9YnkSDLTv8IeTU3tbAvE-1000%254080.jpg
> > > >
> > > > In other cases, I get the same error even though the longest
> filename is less than 100 chars. So that is not the issue...
> > > >
> > > > NOTE: that the error is rare -- I migrated about 2 million pool
> files repeated across 100 backups... yet I only encountered about a dozen
> "unreadable" attrib files
> > > > Also, the "got unreasonable file name length" error is always
> associated with the "can't read attribute file" error (and vice-versa)
> > > >
> > > > I tried to troubleshoot, but the problem seems to be in the
> > > > BackupPC::XS:Attrib read C-code...
> > > >
> > >
> > > OK... I dug into the C-code a bit...
> > >
> > > The problem occurs in bpc_attrib.c in the lines:
> > > } else if ( magic == BPC_ATTRIB_TYPE_UNIX ) {
> > > while ( bufP < buf + nRead ) {
> > > uint fileNameLen;
> > > char *fileName;
> > > bpc_attrib_file *file;
> > > int64 sizeDiv4GB;
> > > uint type;
> > >
> > > if ( nRead == sizeof(buf) && bufP > buf + nRead - 2 *
> BPC_MAXPATHLEN
> > > && read_more_data(&fd, buf, sizeof(buf), &nRead,
> &bufP, attribPath) ) {
> > > bpc_fileZIO_close(&fd);
> > > printf("JJK: magic 2\n");
> > > return -1;
> > > }
> > >
> > > fileNameLen = getVarInt(&bufP, buf + nRead);
> > > if ( fileNameLen > 2 * BPC_MAXPATHLEN - 16 ) {
> > > bpc_logErrf("bpc_attrib_dirRead: got unreasonable
> file name length %d\n", fileNameLen);
> > > bpc_fileZIO_close(&fd);
> > > return -1;
> > > }
> > >
> > >
> > > For example, fileNameLen comes back as 852354 which is obviously a
> ridiculous number.
> > >
> > > I also played with the attrib file itself and found the following:
> > > Specifically, when I break the attrib file into parts:
> > > 1. Every file is individually readable when broken into separate
> attrib files with just one entry
> > > 2. Only certain combinations of files reconstructed into an attrib
> file are not readable. For example, in one case, files 7-118 are not
> readable but files 6-117 or 8-118 are readable.
> > > 2. In v3.x is always readable.
> > >
> > > I suspect that Perl is storing longer hashes in a way that is not
> readable by the C-code in v4.x.
> > >
> > > Happy to share the above examples if anyone is willing to help me
> troubleshoot...
> > >
> > > Jeff
> >
> > Here is a test attrib hash that is readable in 3.x but not 4.x
> >
> > $VAR1 = {
> > 'temp' => {
> > 'uid' => 1023,
> > 'mtime' => 1386359561,
> > 'sizeDiv4GB' => 0,
> > 'size' => 24090,
> > 'sizeMod4GB' => 24090,
> > 'mode' => 33204,
> > 'gid' => 1023,
> > 'type' => 0
> > },
> > 'c.shld.net
> %2Frpx%2Fi%2Fs%2Fpi%2Fmp%2F3962%2F7417620111p%3Fsrc%3Dhttp%253A%252F%
> 252Fwww.petraimages.com%252F500x500%252FBSH260224.jpg' => {
> > 'mode' => 33204,
> > 'gid' => 1023,
> > 'type' => 0,
> > 'mtime' => 1385712394,
> > 'uid' => 1023,
> > 'sizeDiv4GB' => 0,
> > 'size' => 0,
> > 'sizeMod4GB' => 3330
> > }
> > };
> >
> >
> > Perhaps the problem is with the non-ascii encoding in the second file
> > that causes the filename read to fail in the c-code in attrib.c??
> >
>
> I believe the problem is that getVarInt is used instead of
> getVarInt_v3 in bpc_attrib_dirRead in places where we are reading a v3
> rather than v4 file.
> Specifically:
> } else if ( magic == BPC_ATTRIB_TYPE_UNIX ) {
> while ( bufP < buf + nRead ) {
> uint fileNameLen;
> char *fileName;
> bpc_attrib_file *file;
> int64 sizeDiv4GB;
> uint type;
>
> if ( nRead == sizeof(buf) && bufP > buf + nRead - 2 *
> BPC_MAXPATHLEN
> && read_more_data(&fd, buf, sizeof(buf), &nRead,
> &bufP, attribPath) ) {
> bpc_fileZIO_close(&fd);
> return -1;
> }
>
> fileNameLen = getVarInt(&bufP, buf + nRead);
> if ( fileNameLen > 2 * BPC_MAXPATHLEN - 16 ) {
> bpc_logErrf("bpc_attrib_dirRead: got unreasonable file
> name length %d\n", fileNameLen);
> bpc_fileZIO_close(&fd);
> return -1;
> }
>
>
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
