Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Junio C Hamano]

On Thu, Aug 15, 2013 at 2:32 AM, Ben Tebulin  wrote:
> Just as a catchup for everybody being interested:
>
> I finally wrote to the linux-mm newsgroup and Linus pointed out, that
> this might be a known bug yet not fixed in mainline.
>
> Unfortunately this doesn't seem to stand the test; but as far as Git is
> concerned, it appears that that they are willing to take actions.
>
> http://thread.gmane.org/gmane.linux.kernel/1541707/focus=105432

Thanks for following this up.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Ben Tebulin]

Just as a catchup for everybody being interested:

I finally wrote to the linux-mm newsgroup and Linus pointed out, that
this might be a known bug yet not fixed in mainline.

Unfortunately this doesn't seem to stand the test; but as far as Git is
concerned, it appears that that they are willing to take actions.

http://thread.gmane.org/gmane.linux.kernel/1541707/focus=105432




--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Ben Tebulin]

> My best theory so far:
> malloc()/free() actually use mmap()/munmap() for large allocations.
> mallopt(3) tells me that [...]

Many things I don't grab at first go :-)

Last night I did a long git-bisect of the kernel and was able to
pinpoint a change in the Linux memory management as cause (see my
pre-prepared email I just sent).

Would that fit into your theory?

> You could try to gather some data on this by  [...]
Before I dive into that maybe you could throw a quick look at the
causing Kernel change (which reading from the words _could_ be related)
if this fits into the picture?

If yes, I'm happy to try to assist you in further pinning this issue down!
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Thomas Rast]

Ben Tebulin  writes:

> Hello everybody!
>
> I have some _very interesting_ news regarding this issue!
> Here is the deal:
>
>   1. I was able to *reproduce the error on a machine of a coworker!*
>
>   2. I was able to rule out
>   - HDD: It's reproducible from /dev/shm
>   - Memory: Memory tests works fine
>
> now the interesting part:
>
>   3. Occurs on Linux kernels 3.7.10, 3.8.x, 3.9.11, 3.10.5
>  _but not on:_ 3.6.11, 3.5.7 and 2.6.32
>
> Both machines showing this problem are »Dell Latitude E6330« with an
> »i5-3340M @ 2.70GHz« CPU. Mine running stock kernels, coworker using
> plain Ubuntu 13.04.
>
> Furthermore I need to stress, that we never had any issues with our
> devices during daily _at all_.
>
> So what to do best now?

If you're still interested...

What did memcheck86 say about these machines?


My best theory so far:

malloc()/free() actually use mmap()/munmap() for large allocations.
mallopt(3) tells me that "large" here means a dynamic limit on modern
glibc, but with an upper limit of 32MB on 64-bit.  So your 39MB blob
would always result in an mmap() type allocation.

And it should be easy for glibc to check mmap() type allocations against
double free(), at least if the address has not been mapped again in the
same process.  So it probably does that, which aborts the program.

You could try to gather some data on this by looking at

  strace -o fsck.trace -f git fsck

for a crash case.  If the theory is correct, you should see an munmap()
for an address similar to the one pointed out by the glibc error
message, as follows: if you get

  *** glibc detected *** : free(): invalid pointer: 0x7fe129669010 
***

check for the address 0x7fe129669000 (the start of the page that the
address is in).

Note that this is a pretty weak theory, because double free() detection
is squarely up valgrind's alley.  So valgrind's lack of complaints is a
strong argument against it.  But it's the best I've got so far.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Stefan Beller]

On 08/09/2013 02:27 PM, Ben Tebulin wrote:
> Hello everybody!
> 
> I have some _very interesting_ news regarding this issue!
> Here is the deal:
> 
>   1. I was able to *reproduce the error on a machine of a coworker!*
> 
>   2. I was able to rule out
>   - HDD: It's reproducible from /dev/shm
>   - Memory: Memory tests works fine
> 
> now the interesting part:
> 
>   3. Occurs on Linux kernels 3.7.10, 3.8.x, 3.9.11, 3.10.5
>  _but not on:_ 3.6.11, 3.5.7 and 2.6.32
> 
> Both machines showing this problem are »Dell Latitude E6330« with an
> »i5-3340M @ 2.70GHz« CPU. Mine running stock kernels, coworker using
> plain Ubuntu 13.04.
> 
> Furthermore I need to stress, that we never had any issues with our
> devices during daily _at all_.
> 
> So what to do best now?

Maybe include the kernel mailing list, they know what's up ;)
Linking to the start of the discussion $gmane/33602
Maybe you could also run git through strace and diff the output of a
faulty machine with a correct machine?

Stefan




signature.asc
Description: OpenPGP digital signature

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Ben Tebulin]

Hello everybody!

I have some _very interesting_ news regarding this issue!
Here is the deal:

  1. I was able to *reproduce the error on a machine of a coworker!*

  2. I was able to rule out
  - HDD: It's reproducible from /dev/shm
  - Memory: Memory tests works fine

now the interesting part:

  3. Occurs on Linux kernels 3.7.10, 3.8.x, 3.9.11, 3.10.5
 _but not on:_ 3.6.11, 3.5.7 and 2.6.32

Both machines showing this problem are »Dell Latitude E6330« with an
»i5-3340M @ 2.70GHz« CPU. Mine running stock kernels, coworker using
plain Ubuntu 13.04.

Furthermore I need to stress, that we never had any issues with our
devices during daily _at all_.

So what to do best now?

- Ben


Am 08.08.2013 19:38, schrieb Junio C Hamano:
 Can you try to reproduce with a version older than v1.8.3?
 E.g. v1.8.2.3.
>>>
>>> It seems to run 'much better' 
>>>   v1.8.2.3 : 3/10 runs do fail
>>>   fb56570  : 9/10 runs do fail
[..]
>>> They always fail on a big blob (39MB) as I wrote in my first e-mail:
>>>
>>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
>>> 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
>>> error: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978
>>> fatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978
>>> 0
>>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
>>> 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
>>> 39517156
> 
> Hmm, from this, and a later one ...
> 
> ... I am reading that (1) the packfile and repository is basically
> OK, (2) reading that object sometimes fails, and (3) the symptom is
> not limited to fsck but anything that reads the object with
> parse_object().  And that symptom exists only on that single machine
> (I am assuming that the repository was bit-for-bit copied, not
> "cloned", for the purpose of testing it on the other machine).  That
> makes me suspect something outside the control of Git (e.g. faulty
> memory or disk controller cable).
> 
> Are there other big blobs in the repository, and would "show | wc" fail
> if you attempt to read it on that machine?
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20) (gitml: message 6 of 20)

[Junio C Hamano]

Thomas Rast  writes:

> Ben Tebulin  writes:
>
>> Am 08.08.2013 16:20, schrieb Thomas Rast - tr...@inf.ethz.ch:
>>> Can you try to reproduce with a version older than v1.8.3?
>>> E.g. v1.8.2.3.
>>> I'm asking because the above points at packed_object_info(), which I
>>> recently rewrote to be nonrecursive.
>>
>> It seems to run 'much better' 
>>   v1.8.2.3 : 3/10 runs do fail
>>   fb56570  : 9/10 runs do fail
>
> The good news is that this shifts the blame away from my commit ;-) as
> the problem clearly existed even before that.
>
> The bad news, of course, is that this is another hunch that turned out
> to be wrong.  I'm running out of ideas.
>
>> They always fail on a big blob (39MB) as I wrote in my first e-mail:
>>
>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
>> 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
>> error: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978
>> fatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978
>> 0
>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
>> 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
>> 39517156

Hmm, from this, and a later one ...

> Ben Tebulin  writes:
>
> I was unable to reproduce the error with the same repo and same Git
> version on a different machine (Debian Squeeze x64 on a AMD Phenom x6
> 1045T).

... I am reading that (1) the packfile and repository is basically
OK, (2) reading that object sometimes fails, and (3) the symptom is
not limited to fsck but anything that reads the object with
parse_object().  And that symptom exists only on that single machine
(I am assuming that the repository was bit-for-bit copied, not
"cloned", for the purpose of testing it on the other machine).  That
makes me suspect something outside the control of Git (e.g. faulty
memory or disk controller cable).

Are there other big blobs in the repository, and would "show | wc" fail
if you attempt to read it on that machine?

Thanks all for helping to track this, by the way.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Matthieu Moy]

Ben Tebulin  writes:

> c) or defect/buggy Hardware (CPU, Memory)

You may want to try a memcheck86 on your machine for a few hours, in
case ... (if your RAM is broken, you do want to know it!).

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Ben Tebulin]

I was unable to reproduce the error with the same repo and same Git
version on a different machine (Debian Squeeze x64 on a AMD Phenom x6
1045T).


> I'm running out of ideas.
Me, too. Based on out current observations I'd assume one of:

a) a rare, timing-sensitive bug in Git
b) a compiler/distribution/environment sensitive issue
c) or defect/buggy Hardware (CPU, Memory)

> Hrmm.  I wonder about the significance of those 39MB.  What is your
> core.packedGitWindowSize?  (Judging from the pastes you seem to be on
> 64bit, so the default would be 1GB.)
The default. I have 12GB RAM.

So I'll try to rule out b) and c) as far as I can and report in if I
have any new findings.

Nevertheless thank you very much for your extensive assistance!

- Ben
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20) (gitml: message 6 of 20)

[Thomas Rast]

Ben Tebulin  writes:

> Am 08.08.2013 16:20, schrieb Thomas Rast - tr...@inf.ethz.ch:
>> Can you try to reproduce with a version older than v1.8.3?
>> E.g. v1.8.2.3.
>> I'm asking because the above points at packed_object_info(), which I
>> recently rewrote to be nonrecursive.
>
> It seems to run 'much better' 
>   v1.8.2.3 : 3/10 runs do fail
>   fb56570  : 9/10 runs do fail

The good news is that this shifts the blame away from my commit ;-) as
the problem clearly existed even before that.

The bad news, of course, is that this is another hunch that turned out
to be wrong.  I'm running out of ideas.

> They always fail on a big blob (39MB) as I wrote in my first e-mail:
>
> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
> 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
> error: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978
> fatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978
> 0
> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
> 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
> 39517156

Hrmm.  I wonder about the significance of those 39MB.  What is your
core.packedGitWindowSize?  (Judging from the pastes you seem to be on
64bit, so the default would be 1GB.)

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20) (gitml: message 6 of 20)

[Ben Tebulin]

Am 08.08.2013 16:20, schrieb Thomas Rast - tr...@inf.ethz.ch:
> Can you try to reproduce with a version older than v1.8.3?
> E.g. v1.8.2.3.
> I'm asking because the above points at packed_object_info(), which I
> recently rewrote to be nonrecursive.

It seems to run 'much better' 
  v1.8.2.3 : 3/10 runs do fail
  fb56570  : 9/10 runs do fail

They always fail on a big blob (39MB) as I wrote in my first e-mail:

ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
error: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978
fatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978
0
ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 
49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c
39517156


> Also, can you please stop losing the Cc list?  
I'm _very_ sorry for this. I was trying to hide my e-mail 
address from spam robots using spamgourmet.com as remailer.
Unfortunately it breaks CC.  Switching e-mail now.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20)

[Thomas Rast]

gitml.jexp...@recursor.net writes:

>> Using
>> 
>>   addr2line -e ~/projects/git.git/git-fsck
>> 
>> on these addresses may help a little, but not sure it's going to be
>> sufficient :-(.
>
>
> I'm still trying to reproduce this issue using gdb.
> Also I'm trying to reproduce this issue with my git repo on another machine.
>
> ben@n179 /tmp $ addr2line -e ~/projects/git.git/git-fsck
> 0x51e401
> 0x51e53c
> 0x51ecc3
> 0x4e707b
> 0x4e7485
> 0x43d433
> 0x405158
> 0x4052ee
> 0x4054ba
> /home/ben/projects/git.git/sha1_file.c:1901
> /home/ben/projects/git.git/sha1_file.c:1928
> /home/ben/projects/git.git/sha1_file.c:2096
> /home/ben/projects/git.git/pack-check.c:119
> /home/ben/projects/git.git/pack-check.c:177
> /home/ben/projects/git.git/builtin/fsck.c:678
> /home/ben/projects/git.git/git.c:291
> /home/ben/projects/git.git/git.c:453
> /home/ben/projects/git.git/git.c:543

Can you try to reproduce with a version older than v1.8.3?
E.g. v1.8.2.3.

I'm asking because the above points at packed_object_info(), which I
recently rewrote to be nonrecursive.


Also, can you please stop losing the Cc list?  The etiquette on this
list is to Cc everyone who was involved so far, usually meaning everyone
who was already a recipient of the mail you are replying to.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20)

[gitml . jexpert]

> Using
> 
>   addr2line -e ~/projects/git.git/git-fsck
> 
> on these addresses may help a little, but not sure it's going to be
> sufficient :-(.


I'm still trying to reproduce this issue using gdb.
Also I'm trying to reproduce this issue with my git repo on another machine.

ben@n179 /tmp $ addr2line -e ~/projects/git.git/git-fsck
0x51e401
0x51e53c
0x51ecc3
0x4e707b
0x4e7485
0x43d433
0x405158
0x4052ee
0x4054ba
/home/ben/projects/git.git/sha1_file.c:1901
/home/ben/projects/git.git/sha1_file.c:1928
/home/ben/projects/git.git/sha1_file.c:2096
/home/ben/projects/git.git/pack-check.c:119
/home/ben/projects/git.git/pack-check.c:177
/home/ben/projects/git.git/builtin/fsck.c:678
/home/ben/projects/git.git/git.c:291
/home/ben/projects/git.git/git.c:453
/home/ben/projects/git.git/git.c:543

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[Matthieu Moy]

gitml.jexp...@recursor.net writes:

> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fe162d3eb96]
> [0x51e401]
> [0x51e53c]
> [0x51ecc3]
> [0x4e707b]
> [0x4e7485]
> [0x43d433]
> [0x405158]
> [0x4052ee]
> [0x4054ba]

Using

  addr2line -e ~/projects/git.git/git-fsck

on these addresses may help a little, but not sure it's going to be
sufficient :-(.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message

[gitml . jexpert]

Am 08.08.2013 15:18, schrieb Matthieu Moy - matthieu@grenoble-inp.fr:> 
gitml.jexp...@recursor.net writes:
>> So - now the puzzling thing: With valgrind it seems to work! 
> Weird, indeed. What about GDB ?


Ah - come on. Is this another Heisenberg bug ?
Still trying to reproduce it using gdb and/or valgrind...



ben@n179 /tmp/project.git $ gdb --args ~/projects/git.git/git-fsck
GNU gdb (GDB) 7.5-ubuntu
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /home/ben/projects/git.git/git-fsck...done.
(gdb) run
Starting program: /home/ben/projects/git.git/git-fsck 
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x77ffa000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Checking object directories: 100% (256/256), done.
Checking objects: 100% (63588/63588), done.
[Inferior 1 (process 3795) exited normally]
(gdb) quit

ben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck
Checking object directories: 100% (256/256), done.
error: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from 
.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt
Checking objects: 100% (63588/63588), done.

ben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck
Checking object directories: 100% (256/256), done.
error: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from 
.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt
Checking objects: 100% (63588/63588), done.

ben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (63588/63588), done.

ben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck
Checking object directories: 100% (256/256), done.
error: packed 0f5f33639bfc1a781fe080c31a1f076d9a25c1d3 from 
.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt
*** glibc detected *** : free(): invalid pointer: 0x7fe129669010 
***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fe162d3eb96]
[0x51e401]
[0x51e53c]
[0x51ecc3]
[0x4e707b]
[0x4e7485]
[0x43d433]
[0x405158]
[0x4052ee]
[0x4054ba]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fe162ce176d]
[0x404429]
=== Memory map: 
0040-005bb000 r-xp  fc:01 3939047
/home/ben/projects/git.git/git-fsck
007ba000-007bb000 r--p 001ba000 fc:01 3939047
/home/ben/projects/git.git/git-fsck
007bb000-007c2000 rw-p 001bb000 fc:01 3939047
/home/ben/projects/git.git/git-fsck
007c2000-00822000 rw-p  00:00 0 
0277-05809000 rw-p  00:00 0  [heap]
7fe129669000-7fe12bc38000 rw-p  00:00 0 
7fe12dff5000-7fe12e00a000 r-xp  fc:01 7081526
/lib/x86_64-linux-gnu/libgcc_s.so.1
7fe12e00a000-7fe12e209000 ---p 00015000 fc:01 7081526
/lib/x86_64-linux-gnu/libgcc_s.so.1
7fe12e209000-7fe12e20a000 r--p 00014000 fc:01 7081526
/lib/x86_64-linux-gnu/libgcc_s.so.1
7fe12e20a000-7fe12e20b000 rw-p 00015000 fc:01 7081526
/lib/x86_64-linux-gnu/libgcc_s.so.1
7fe12e20b000-7fe13e9c1000 r--p  fc:01 10879034   
/tmp/project.git/.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack
7fe13e9c1000-7fe162a0b000 r--p  fc:01 10881032   
/tmp/project.git/.git/objects/pack/pack-5b132454d5acb542969a939fe55186a65d9dd697.pack
7fe162a0b000-7fe162abc000 r--p  fc:01 10902665   
/tmp/project.git/.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.idx
7fe162abc000-7fe162abe000 r-xp  fc:01 7085848
/lib/x86_64-linux-gnu/libdl-2.15.so
7fe162abe000-7fe162cbe000 ---p 2000 fc:01 7085848
/lib/x86_64-linux-gnu/libdl-2.15.so
7fe162cbe000-7fe162cbf000 r--p 2000 fc:01 7085848
/lib/x86_64-linux-gnu/libdl-2.15.so
7fe162cbf000-7fe162cc rw-p 3000 fc:01 7085848
/lib/x86_64-linux-gnu/libdl-2.15.so
7fe162cc-7fe162e75000 r-xp  fc:01 7085850
/lib/x86_64-linux-gnu/libc-2.15.so
7fe162e75000-7fe163074000 ---p 001b5000 fc:01 7085850
/lib/x86_64-linux-gnu/libc-2.15.so
7fe163074000-7fe163078000 r--p 001b4000 fc:01 7085850
/lib/x86_64-linux-gnu/libc-2.15.so
7fe163078000-7fe16307a000 rw-p 001b8000 fc:01 7085850
/lib/x86_64-linux-gnu/libc-2.15.so

Re: Reproducible, corrupt packfile after fresh git-svn checkout message 3 of 20)

[Thomas Rast]

gitml.jexp...@recursor.net writes:

> So - now the puzzling thing: With valgrind it seems to work! 
> If I run it plain, it doesn't:
>
> /tmp/project.git $ valgrind --track-origins=yes  ~/projects/git.git/git-fsck
[...]
> ==3431== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
> /tmp/project.git $ ~/projects/git.git/git-fsck
> Checking object directories: 100% (256/256), done.
> error: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from
> .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack
> is corrupt
> *** glibc detected *** : free(): invalid pointer: 0x7f8576682010 
> ***
> === Backtrace: =
> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f85ad765b96]
> [0x4e727c]
[...]
> Any further hints?

Hrm.  Can you try getting a backtrace from

  $ gdb ~/projects/git.git/git-fsck
  (gdb) run
  ... wait until it crashes ...
  (gdb) backtrace full

I would have been more interested in error output from valgrind, because
memory corruption invariably happens long before glibc finally figures
out that something is amiss.  But as things stand, the backtrace is
probably the only thing we have...

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message 3 of 20)

[Matthieu Moy]

gitml.jexp...@recursor.net writes:

> So - now the puzzling thing: With valgrind it seems to work! 

Weird, indeed. What about GDB ?

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout message 3 of 20)

[gitml . jexpert]

Great! Thank you all guys for your extensive instructions!

@Thomas: I only had to add libexpat1-dev to the list.

I checked out Git v1.8.3.4 as this was my used verion and built as
instructed. The error is still reproducible using the "CFLAGS = -O0 -g"
build.

So - now the puzzling thing: With valgrind it seems to work! 
If I run it plain, it doesn't:

/tmp/project.git $ valgrind --track-origins=yes  ~/projects/git.git/git-fsck
==3431== Memcheck, a memory error detector
==3431== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3431== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3431== Command: /home/ben/projects/git.git/git-fsck
==3431== 
==3431== Warning: set address range perms: large range [0x70b5000, 0x2b0ff000) 
(defined)
Checking object directories: 100% (256/256), done.
==3431== Warning: set address range perms: large range [0x3959d000, 0x49d53000) 
(defined)
Checking objects: 100% (63588/63588), done.
Checking connectivity: 68696, done.
==3431== 
==3431== HEAP SUMMARY:
==3431== in use at exit: 7,574,911 bytes in 15,428 blocks
==3431==   total heap usage: 422,498 allocs, 407,070 frees, 4,396,939,465 bytes 
allocated
==3431== 
==3431== LEAK SUMMARY:
==3431==definitely lost: 0 bytes in 0 blocks
==3431==indirectly lost: 0 bytes in 0 blocks
==3431==  possibly lost: 0 bytes in 0 blocks
==3431==still reachable: 7,574,911 bytes in 15,428 blocks
==3431== suppressed: 0 bytes in 0 blocks
==3431== Rerun with --leak-check=full to see details of leaked memory
==3431== 
==3431== For counts of detected and suppressed errors, rerun with: -v
==3431== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
/tmp/project.git $ ~/projects/git.git/git-fsck
Checking object directories: 100% (256/256), done.
error: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from 
.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt
*** glibc detected *** : free(): invalid pointer: 0x7f8576682010 
***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f85ad765b96]
[0x4e727c]


I'm using a i5-3340M on Linux Mint 14 with a offical Linux 3.9.9
gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2

Any further hints?


> Try something like
>
>   # The package names might be wrong and/or you may need additional -dev
>   # packages, I don't know the specifics for ubuntu
>   apt-get install gcc make valgrind libcurl-dev zlib-dev
>   cd
>   git clone git://git.kernel.org/pub/scm/git/git.git
>   cd git
>   echo 'CFLAGS = -O0 -g' >>config.mak
>   make
>   cd /path/to/repo
>   valgrind --track-origins=yes ~/git/git-fsck
>
> It'll be very slow, at least 20x the normal runtime, so don't be
> surprised if it doesn't seem to get anywhere at first.
>

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout

[Stefan Beller]

On 08/08/2013 02:23 PM, gitml.jexp...@recursor.net wrote:
>> Regardless of any possible fault in git-svn, there's an obvious bug here
>> with git-fsck.  Can you share the pack (if the project is public) or
>> compile a git-fsck without optimization and with debugging, and run it
>> under valgrind, to hopefully get us a backtrace of where the memory
>> management goes off the rails?
> 
> Unfortunately I'm unable to share the pack.
> 
> As Java Developer I'm note very savy, but I'd try.
> Do you have me any pointers on ".. without optimization and with
> debugging" and "run it under valgrind"?
> 
> Currently I used
>deb http://ppa.launchpad.net/git-core/ppa/ubuntu quantal main
> as source.

The version from the ppa seems to be 1.7.0.4-1ubuntu0.2 but the version
of git being in the native ubuntu quantal repository 
(I assume you're running Ubuntu?) is already 1.7.10.4-1ubuntu1 
(http://packages.ubuntu.com/quantal/git)


However to get an unoptimized version with debug symbols you could do this:

# get the source:
git clone git://git.kernel.org/pub/scm/git/git.git/
cd git

# read INSTALL, the section starting at 
# > Git is reasonably self-sufficient, but does depend on a few external
# > programs and libraries
# this is a guess, maybe you need more or less packages, but you'd need 
the -dev packages for the C-headers
sudo apt-get install libz-dev libopenssl-dev libcurl-dev libexpat-dev 

# Now disable optimizing by overwriting default compile flags:
echo "CFLAGS=-g" > config.mak

# compile and install   
make 
make install # installs to ~/bin

# open a new shell or relogin, and see if you're using the version you 
just build:
git --version
# git version 1.8.4.rc1

Stefan



signature.asc
Description: OpenPGP digital signature

Re: Reproducible, corrupt packfile after fresh git-svn checkout

[Matthieu Moy]

gitml.jexp...@recursor.net writes:

>> Regardless of any possible fault in git-svn, there's an obvious bug here
>> with git-fsck.  Can you share the pack (if the project is public) or
>> compile a git-fsck without optimization and with debugging, and run it
>> under valgrind, to hopefully get us a backtrace of where the memory
>> management goes off the rails?
>
> Unfortunately I'm unable to share the pack.
>
> As Java Developer I'm note very savy, but I'd try.
> Do you have me any pointers on ".. without optimization and with
> debugging" and "run it under valgrind"?

To fetch sources and compile:

  git clone https://github.com/git/git.git
  cd git/
  make CFLAGS='-g -O0'

To run gdb:

  gdb --args ./git fsck

Then type "run" at the gdb prompt. When the program crashes, you should
get the gdb prompt back. Then, typing "bt" gives you a backtrace, and
"bt full" gives you the same with more information (be careful, there
may be private information in it).

To use valgrind, just do:

  valgrind ./git fsck

(essentially, it runs the program adding a bunch of runtime checks hence
may provide better diagnosis)

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout

[Thomas Rast]

gitml.jexp...@recursor.net writes:

>> Regardless of any possible fault in git-svn, there's an obvious bug here
>> with git-fsck.  Can you share the pack (if the project is public) or
>> compile a git-fsck without optimization and with debugging, and run it
>> under valgrind, to hopefully get us a backtrace of where the memory
>> management goes off the rails?
>
> Unfortunately I'm unable to share the pack.
>
> As Java Developer I'm note very savy, but I'd try.
> Do you have me any pointers on ".. without optimization and with
> debugging" and "run it under valgrind"?
>
> Currently I used
>deb http://ppa.launchpad.net/git-core/ppa/ubuntu quantal main
> as source.

Try something like

  # The package names might be wrong and/or you may need additional -dev
  # packages, I don't know the specifics for ubuntu
  apt-get install gcc make valgrind libcurl-dev zlib-dev
  cd
  git clone git://git.kernel.org/pub/scm/git/git.git
  cd git
  echo 'CFLAGS = -O0 -g' >>config.mak
  make
  cd /path/to/repo
  valgrind --track-origins=yes ~/git/git-fsck

It'll be very slow, at least 20x the normal runtime, so don't be
surprised if it doesn't seem to get anywhere at first.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reproducible, corrupt packfile after fresh git-svn checkout

[gitml . jexpert]

> Regardless of any possible fault in git-svn, there's an obvious bug here
> with git-fsck.  Can you share the pack (if the project is public) or
> compile a git-fsck without optimization and with debugging, and run it
> under valgrind, to hopefully get us a backtrace of where the memory
> management goes off the rails?

Unfortunately I'm unable to share the pack.

As Java Developer I'm note very savy, but I'd try.
Do you have me any pointers on ".. without optimization and with
debugging" and "run it under valgrind"?

Currently I used
   deb http://ppa.launchpad.net/git-core/ppa/ubuntu quantal main
as source.

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html