Re: git checkout crashes after server being updated to Debian X86_64

[Duy Nguyen]

On Tue, Oct 18, 2016 at 10:17 PM, Raffael Reichelt
 wrote:
> Hello!
>
> I have a serious problem with git, After my provider had updated to a X86_64 
> architecture git crashes with various memory-related errors. This is 
> happening remote when pushing to the repository from my local machine as well 
> as trying it on a shell on the server itself.
>
> This are the error-messages:
>
> fatal: Out of memory, realloc failed
> fatal: recursion detected in die handler
> fatal: recursion detected in die handler

You other mail said memory is capped at 600MB, which should be a lot
for normal repositories. If you set the environment variable
GIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes
first) and git attempts to allocate more than that (just that one
time, not total mem) then it's caught and we get a glimpse of how much
memory git may need. Unfortunately we can't get a stack trace or
anything like that unless you rebuild Git from source.

> or
> fatal: unable to create threaded lstat
> fatal: recursion detected in die handler

Hmm.. with "max user processes (-u) 42" we should be fine because we
only create 20 threads max. What happens if you set core.preloadindex
to false? Can it run until the end or hit some other fatal errors?

There's room for improvement in preload_index(). If we hit resource
limit like this, it's not the end of the world and we should be able
to keep going. But threaded lstat has been available for a long time
and this is the first time I see a report like this, not sure if it's
worth fixing.
-- 
Duy

Re: git checkout crashes after server being updated to Debian X86_64

[Raffael Reichelt]

> Am 19.10.2016 um 15:27 schrieb Duy Nguyen :
> 
> On Tue, Oct 18, 2016 at 10:17 PM, Raffael Reichelt
>  wrote:
>> Hello!
>> 
>> I have a serious problem with git, After my provider had updated to a X86_64 
>> architecture git crashes with various memory-related errors. This is 
>> happening remote when pushing to the repository from my local machine as 
>> well as trying it on a shell on the server itself.
>> 
>> This are the error-messages:
>> 
>> fatal: Out of memory, realloc failed
>> fatal: recursion detected in die handler
>> fatal: recursion detected in die handler
> 
> You other mail said memory is capped at 600MB, which should be a lot
> for normal repositories. If you set the environment variable
> GIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes
> first) and git attempts to allocate more than that (just that one
> time, not total mem) then it's caught and we get a glimpse of how much
> memory git may need. Unfortunately we can't get a stack trace or
> anything like that unless you rebuild Git from source.

This was no change: crashed with the same errors …

> 
>> or
>> fatal: unable to create threaded lstat
>> fatal: recursion detected in die handler
> 
> Hmm.. with "max user processes (-u) 42" we should be fine because we
> only create 20 threads max. What happens if you set core.preloadindex
> to false? Can it run until the end or hit some other fatal errors?
> 

This did the trick :) I just repeatedly did a forced checkout and it went until 
the end without errors
THX a lot!

Raffael

Re: git checkout crashes after server being updated to Debian X86_64

[Raffael Reichelt]

Hello Renè!
file is returning 

/usr/bin/git: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically 
linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32,  
BuildID[sha1]=ee62e538d6fe6673d3ba49f0e66bfec784cc32bc, stripped

and ulimit is:
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 1
file size   (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files  (-n) 512
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) 1800
max user processes  (-u) 42
virtual memory  (kbytes, -v) 786432
file locks  (-x) unlimited

Support told me git is limited to 600M

Regrads,
Rafael

> Am 18.10.2016 um 18:42 schrieb René Scharfe :
> 
> Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:
>> Hello!
>> 
>> I have a serious problem with git, After my provider had updated to a
>> X86_64 architecture git crashes with various memory-related errors.
>> This is happening remote when pushing to the repository from my local
>> machine as well as trying it on a shell on the server itself.
>> 
>> This are the error-messages:
>> 
>> fatal: Out of memory, realloc failed
>> fatal: recursion detected in die handler
>> fatal: recursion detected in die handler
>> 
>> or
>> fatal: unable to create threaded lstat
>> fatal: recursion detected in die handler
>> or
>> fatal: unable to create threaded lstat
>> *** Error in `git': double free or corruption (fasttop): 0x00a8ade0 
>> ***
>> fatal: recursion detected in die handler
>> Aborted
>> 
>> It’s obviously not a problem of the repository - happens with all of
>> them. I think it is also not a question of size - happens with a 80M
>> Repository as well as with a 500M one.
>> 
>> Any way: did a
>> 
>> git fsck
>> Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.
>> Prüfe Objekte: 100% (56305/56305), Fertig.
>> 
>> git gc --auto --prune=today —aggressive
>> git repack
>> 
>> Additionally I played around some config parameters  so my config now looks 
>> like:
>> [http]
>>postbuffer = 524288000
>> [pack]
>>threads = 1
>>deltaCacheSize = 128m
>>packSizeLimit = 128m
>>windowMemory = 128m
>> [core]
>>packedGitLimit = 128m
>>packedGitWindowSize = 128m
>>repositoryformatversion = 0
>>filemode = true
>>bare = true
>> 
>> I am running
>> git version 2.1.4
>> 
>> on
>> Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 
>> 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux
>> 
>> Anyone out there to help me getting out of this trouble?
> 
> Git 2.1.4 is the version that comes with Debian stable according to 
> https://packages.debian.org/jessie/git, so I guess using a more recent 
> version is not a reasonable option.
> 
> What do "file $(which git)" and "ulimit -a" return?  Do you have an x86-64 
> binary and no unnecessarily low limits set?
> 
> René

Re: git checkout crashes after server being updated to Debian X86_64

[René Scharfe]

Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:

Hello!

I have a serious problem with git, After my provider had updated to a
X86_64 architecture git crashes with various memory-related errors.
This is happening remote when pushing to the repository from my local
machine as well as trying it on a shell on the server itself.

This are the error-messages:

fatal: Out of memory, realloc failed
fatal: recursion detected in die handler
fatal: recursion detected in die handler

or
fatal: unable to create threaded lstat
fatal: recursion detected in die handler
or
fatal: unable to create threaded lstat
*** Error in `git': double free or corruption (fasttop): 0x00a8ade0 ***
fatal: recursion detected in die handler
Aborted

It’s obviously not a problem of the repository - happens with all of
them. I think it is also not a question of size - happens with a 80M
Repository as well as with a 500M one.

Any way: did a

git fsck
Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.
Prüfe Objekte: 100% (56305/56305), Fertig.

git gc --auto --prune=today —aggressive
git repack

Additionally I played around some config parameters  so my config now looks 
like:
[http]
postbuffer = 524288000
[pack]
threads = 1
deltaCacheSize = 128m
packSizeLimit = 128m
windowMemory = 128m
[core]
packedGitLimit = 128m
packedGitWindowSize = 128m
repositoryformatversion = 0
filemode = true
bare = true

I am running
git version 2.1.4

on
Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 
3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux

Anyone out there to help me getting out of this trouble?


Git 2.1.4 is the version that comes with Debian stable according to 
https://packages.debian.org/jessie/git, so I guess using a more recent 
version is not a reasonable option.


What do "file $(which git)" and "ulimit -a" return?  Do you have an 
x86-64 binary and no unnecessarily low limits set?


René