Re: fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

On Thu, 10 Aug 2017, Jeff King wrote:

> On Thu, Aug 10, 2017 at 09:58:37PM +0200, René Scharfe wrote:

> > > So the function is returning -1 and leaving ENOMEM in errno on
> > > Yaroslav's system.

> > > I wonder if we are truly hitting out of memory, though.  The same
> > > symptom could bee seen if getdelim() does not touch errno when it
> > > returns -1, but some other system call earlier set it to ENOMEM,
> > > for example.

> > That can happen when the end of a file is reached; getdelim() returns
> > -1 and leaves errno unchanged.  So we need to set errno to 0 just
> > before that call.

> Good catch. That's a bug in my original conversoin of
> strbuf_getwholeline().

I think this did it!  at least on this simple test... yet to test a bit
more in those scenarios we ran into it before while testing git-annex.

commit 36ef5e3ad2c187d3be664c33dbc8c06e59bceaf4 (HEAD -> bf-seterrno0)
Author: Yaroslav O. Halchenko 
Date:   Thu Aug 10 20:26:47 2017 +

BF: set errno to 0 before getdelim call to get unbiased assesment later

diff --git a/strbuf.c b/strbuf.c
index 89d22e3b0..323c49ceb 100644
--- a/strbuf.c
+++ b/strbuf.c
@@ -476,6 +476,7 @@ int strbuf_getwholeline(struct strbuf *sb, FILE *fp, int 
term)
/* Translate slopbuf to NULL, as we cannot call realloc on it */
if (!sb->alloc)
sb->buf = NULL;
+   errno = 0;
r = getdelim(>buf, >alloc, term, fp);
 
if (r > 0) {

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: fatal: Out of memory, getdelim failed under NFS mounts

[Jeff King]

On Thu, Aug 10, 2017 at 09:58:37PM +0200, René Scharfe wrote:

> > So the function is returning -1 and leaving ENOMEM in errno on
> > Yaroslav's system.
> > 
> > I wonder if we are truly hitting out of memory, though.  The same
> > symptom could bee seen if getdelim() does not touch errno when it
> > returns -1, but some other system call earlier set it to ENOMEM,
> > for example.
> 
> That can happen when the end of a file is reached; getdelim() returns
> -1 and leaves errno unchanged.  So we need to set errno to 0 just
> before that call.

Good catch. That's a bug in my original conversoin of
strbuf_getwholeline().

-Peff

Re: fatal: Out of memory, getdelim failed under NFS mounts

[René Scharfe]

Am 10.08.2017 um 20:56 schrieb Junio C Hamano:
> René Scharfe  writes:
> 
>> I doubt the type of file system matters.  The questions are: How much
>> main memory do you have, what is git trying to cram into it, is there
>> a way to reduce the memory footprint or do you need to add more RAM?
>>
>>> any recommendations on how to pin point the "offender"? ;)
>> Running "GIT_TRACE=1 git pull --ff-only origin master" would be a
>> good start, I think, to find out which of the different activities
>> that pull is doing causes the out-of-memory error.
>>
>> "free" and "ulimit -a" can help you find out how much memory you can
>> use.
>>
>> Also: What does "wc -L .git/FETCH_HEAD .git/packed-refs" report?
>> getdelim() is used mostly to read lines from files like these and in
>> the admittedly unlikely case that they are *really* long such an
>> error would be expected.
> 
> There is only one getdelim() call, which was introduced in v2.5.0
> timeframe, and it is used like this:
> 
>   r = getdelim(>buf, >alloc, term, fp);
> 
>   if (r > 0) {
>   sb->len = r;
>   return 0;
>   }
>   assert(r == -1);
> 
>   /*
>* Normally we would have called xrealloc, which will try to free
>* memory and recover. But we have no way to tell getdelim() to do so.
>* Worse, we cannot try to recover ENOMEM ourselves, because we have
>* no idea how many bytes were read by getdelim.
>*
>* Dying here is reasonable. It mirrors what xrealloc would do on
>* catastrophic memory failure. We skip the opportunity to free pack
>* memory and retry, but that's unlikely to help for a malloc small
>* enough to hold a single line of input, anyway.
>*/
>   if (errno == ENOMEM)
>   die("Out of memory, getdelim failed");
> 
> So the function is returning -1 and leaving ENOMEM in errno on
> Yaroslav's system.
> 
> I wonder if we are truly hitting out of memory, though.  The same
> symptom could bee seen if getdelim() does not touch errno when it
> returns -1, but some other system call earlier set it to ENOMEM,
> for example.

That can happen when the end of a file is reached; getdelim() returns
-1 and leaves errno unchanged.  So we need to set errno to 0 just
before that call.

Still *some* function must have run into a memory shortage in that
scenario, so adding/assigning more should help, right?

> If the same version of Git is recompiled there without HAVE_GETDELIM
> defined, would it still die with out of memory (presumably inside
> the call to strbuf_grow() in the strbuf_getwholeline() function)?

It's worth a try, if possible.

René

Re: fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

Thank you Junio

On Thu, 10 Aug 2017, Junio C Hamano wrote:
> There is only one getdelim() call, which was introduced in v2.5.0
> timeframe, and it is used like this:

>   r = getdelim(>buf, >alloc, term, fp);

>   if (r > 0) {
>   sb->len = r;
>   return 0;
>   }
>   assert(r == -1);

>   /*
>* Normally we would have called xrealloc, which will try to free
>* memory and recover. But we have no way to tell getdelim() to do so.
>* Worse, we cannot try to recover ENOMEM ourselves, because we have
>* no idea how many bytes were read by getdelim.
>*
>* Dying here is reasonable. It mirrors what xrealloc would do on
>* catastrophic memory failure. We skip the opportunity to free pack
>* memory and retry, but that's unlikely to help for a malloc small
>* enough to hold a single line of input, anyway.
>*/
>   if (errno == ENOMEM)
>   die("Out of memory, getdelim failed");

> So the function is returning -1 and leaving ENOMEM in errno on
> Yaroslav's system.  

> I wonder if we are truly hitting out of memory, though.  The same
> symptom could bee seen if getdelim() does not touch errno when it
> returns -1, but some other system call earlier set it to ENOMEM,
> for example.

> If the same version of Git is recompiled there without HAVE_GETDELIM
> defined, would it still die with out of memory (presumably inside
> the call to strbuf_grow() in the strbuf_getwholeline() function)?

will check now...  for my own education (rotten by Python) -- how
do you know which syscall set errno to be analyzed at this specific
point?  may be it was already set to ENOMEM before entry to this
function?

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: fatal: Out of memory, getdelim failed under NFS mounts

[René Scharfe]

Am 10.08.2017 um 16:43 schrieb Yaroslav Halchenko:
> On Thu, 10 Aug 2017, René Scharfe wrote:
>> Am 09.08.2017 um 19:39 schrieb Yaroslav Halchenko:
>>> More context (may be different issue(s)) could be found at
>>> http://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/
>>> but currently I am consistently reproducing it while running
>>> git (1:2.11.0-3 debian stretch build) within debian stretch singularity
>>> environment [1].
> 
>>> External system is Centos 6.9, and git 1.7.1 (and installed in modules
>>> 2.0.4) do not show similar buggy behavior.
> 
>>> NFS mounted partitions are bind mounted inside the sinularity space and
>>> when I try to do some git operations, I get that error inconsistently , e.g.
> 
>>> yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
>>> master
>>> fatal: Out of memory, getdelim failed
>>> error: git://github.com/datalad/datalad did not send all necessary 
>>> objects
> 
>>> yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
>>> master
>>> fatal: Out of memory, getdelim failed
>>> error: git://github.com/datalad/datalad did not send all necessary 
>>> objects
> 
>>> yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
>>> master
>>> From git://github.com/datalad/datalad
>>>  * branch  master -> FETCH_HEAD
>>> fatal: Out of memory, getdelim failed
> 
>>> and some times it succeeds.  So it smells that some race condition
>>> somewhere...?
> 
>> I doubt the type of file system matters.
> 
> So far it has been a very consistent indicator.  I did not manage to get
> this error while performing the same operation under /tmp (bind to local
> mounted drive), where it also feels going faster (again suggesting that
> original issue is some kind of a race)

Well, there have been bugs in getdelim() before, e.g.:

  https://bugzilla.redhat.com/show_bug.cgi?id=601071
  https://bugzilla.redhat.com/show_bug.cgi?id=1332917

git v2.5.0 was the first version to use it.   So if all else fails it may
be worth compiling git without HAVE_GETDELIM.

>> The questions are: How much
>> main memory do you have, what is git trying to cram into it, is there
>> a way to reduce the memory footprint or do you need to add more RAM?
>> ... reordered ...
>> "free" and "ulimit -a" can help you find out how much memory you can
>> use.
> 
> I think those aren't the reason:
> 
> yhalchen@discovery:/mnt/scratch/yoh/datalad$ free -h
>totalusedfree  shared  buff/cache   
> available
> Mem:   126G2.5G 90G652K 33G
> 123G
> Swap:  127G1.7M127G

Is all of that available to the git in the Singularity container or
is that the memory size of the host and there's some kind of limit
for the guests?

> yhalchen@discovery:/mnt/scratch/yoh/datalad$ ulimit
> unlimited

That's just the maximum file size; memory-related limits are more
interesting for this case.  "ulimit -a" will show all limits.

>>> any recommendations on how to pin point the "offender"? ;)
>> Running "GIT_TRACE=1 git pull --ff-only origin master" would be a
>> good start, I think, to find out which of the different activities
>> that pull is doing causes the out-of-memory error.
> 
> samples of bad, and then good runs (from eyeballing -- the same until
> error message):
> 
> yhalchen@discovery:/mnt/scratch/yoh$ cat git_trace_bad.log
> 14:05:25.782270 git.c:371   trace: built-in: git 'pull' 
> '--ff-only' 'origin' 'master'
> 14:05:25.795036 run-command.c:350   trace: run_command: 'fetch' 
> '--update-head-ok' 'origin' 'master'
> 14:05:25.795332 exec_cmd.c:116  trace: exec: 'git' 'fetch' 
> '--update-head-ok' 'origin' 'master'
> 14:05:25.797212 git.c:371   trace: built-in: git 'fetch' 
> '--update-head-ok' 'origin' 'master'
> 14:05:25.904088 run-command.c:350   trace: run_command: 'rev-list' 
> '--objects' '--stdin' '--not' '--all' '--quiet'
> 14:05:26.085954 run-command.c:350   trace: run_command: 'index-pack' 
> '--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
> discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
> 14:05:26.086333 exec_cmd.c:116  trace: exec: 'git' 'index-pack' 
> '--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
> discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
> 14:05:26.088382 git.c:371   trace: built-in: git 'index-pack' 
> '--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
> discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
> 14:05:26.133326 run-command.c:350   trace: run_command: 'rev-list' 
> '--objects' '--stdin' '--not' '--all' '--quiet'
> 14:05:26.133688 exec_cmd.c:116  trace: exec: 'git' 'rev-list' 
> '--objects' '--stdin' '--not' '--all' '--quiet'
> 14:05:26.135493 git.c:371   trace: built-in: git 'rev-list' 
> '--objects' '--stdin' '--not' '--all' '--quiet'
> fatal: Out of memory, getdelim failed

Re: fatal: Out of memory, getdelim failed under NFS mounts

[Junio C Hamano]

René Scharfe  writes:

> I doubt the type of file system matters.  The questions are: How much
> main memory do you have, what is git trying to cram into it, is there
> a way to reduce the memory footprint or do you need to add more RAM?
>
>> any recommendations on how to pin point the "offender"? ;)
> Running "GIT_TRACE=1 git pull --ff-only origin master" would be a
> good start, I think, to find out which of the different activities
> that pull is doing causes the out-of-memory error.
>
> "free" and "ulimit -a" can help you find out how much memory you can
> use.
>
> Also: What does "wc -L .git/FETCH_HEAD .git/packed-refs" report?
> getdelim() is used mostly to read lines from files like these and in
> the admittedly unlikely case that they are *really* long such an
> error would be expected.

There is only one getdelim() call, which was introduced in v2.5.0
timeframe, and it is used like this:

r = getdelim(>buf, >alloc, term, fp);

if (r > 0) {
sb->len = r;
return 0;
}
assert(r == -1);

/*
 * Normally we would have called xrealloc, which will try to free
 * memory and recover. But we have no way to tell getdelim() to do so.
 * Worse, we cannot try to recover ENOMEM ourselves, because we have
 * no idea how many bytes were read by getdelim.
 *
 * Dying here is reasonable. It mirrors what xrealloc would do on
 * catastrophic memory failure. We skip the opportunity to free pack
 * memory and retry, but that's unlikely to help for a malloc small
 * enough to hold a single line of input, anyway.
 */
if (errno == ENOMEM)
die("Out of memory, getdelim failed");

So the function is returning -1 and leaving ENOMEM in errno on
Yaroslav's system.  

I wonder if we are truly hitting out of memory, though.  The same
symptom could bee seen if getdelim() does not touch errno when it
returns -1, but some other system call earlier set it to ENOMEM,
for example.

If the same version of Git is recompiled there without HAVE_GETDELIM
defined, would it still die with out of memory (presumably inside
the call to strbuf_grow() in the strbuf_getwholeline() function)?

Re: fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

Thank you René!  comments/answers embedded below

On Thu, 10 Aug 2017, René Scharfe wrote:

> Am 09.08.2017 um 19:39 schrieb Yaroslav Halchenko:
> > More context (may be different issue(s)) could be found at
> > http://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/
> > but currently I am consistently reproducing it while running
> > git (1:2.11.0-3 debian stretch build) within debian stretch singularity
> > environment [1].

> > External system is Centos 6.9, and git 1.7.1 (and installed in modules
> > 2.0.4) do not show similar buggy behavior.

> > NFS mounted partitions are bind mounted inside the sinularity space and
> > when I try to do some git operations, I get that error inconsistently , e.g.

> > yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> > master
> > fatal: Out of memory, getdelim failed
> > error: git://github.com/datalad/datalad did not send all necessary 
> > objects

> > yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> > master
> > fatal: Out of memory, getdelim failed
> > error: git://github.com/datalad/datalad did not send all necessary 
> > objects

> > yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> > master
> > From git://github.com/datalad/datalad
> >  * branch  master -> FETCH_HEAD
> > fatal: Out of memory, getdelim failed

> > and some times it succeeds.  So it smells that some race condition
> > somewhere...?

> I doubt the type of file system matters.  

So far it has been a very consistent indicator.  I did not manage to get
this error while performing the same operation under /tmp (bind to local
mounted drive), where it also feels going faster (again suggesting that
original issue is some kind of a race)

> The questions are: How much
> main memory do you have, what is git trying to cram into it, is there
> a way to reduce the memory footprint or do you need to add more RAM?
> ... reordered ...
> "free" and "ulimit -a" can help you find out how much memory you can
> use.

I think those aren't the reason:

yhalchen@discovery:/mnt/scratch/yoh/datalad$ free -h
  totalusedfree  shared  buff/cache   available
Mem:   126G2.5G 90G652K 33G123G
Swap:  127G1.7M127G
yhalchen@discovery:/mnt/scratch/yoh/datalad$ ulimit
unlimited

> > any recommendations on how to pin point the "offender"? ;)
> Running "GIT_TRACE=1 git pull --ff-only origin master" would be a
> good start, I think, to find out which of the different activities
> that pull is doing causes the out-of-memory error.

samples of bad, and then good runs (from eyeballing -- the same until
error message):

yhalchen@discovery:/mnt/scratch/yoh$ cat git_trace_bad.log
14:05:25.782270 git.c:371   trace: built-in: git 'pull' '--ff-only' 
'origin' 'master'
14:05:25.795036 run-command.c:350   trace: run_command: 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:25.795332 exec_cmd.c:116  trace: exec: 'git' 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:25.797212 git.c:371   trace: built-in: git 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:25.904088 run-command.c:350   trace: run_command: 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
14:05:26.085954 run-command.c:350   trace: run_command: 'index-pack' 
'--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
14:05:26.086333 exec_cmd.c:116  trace: exec: 'git' 'index-pack' 
'--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
14:05:26.088382 git.c:371   trace: built-in: git 'index-pack' 
'--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
14:05:26.133326 run-command.c:350   trace: run_command: 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
14:05:26.133688 exec_cmd.c:116  trace: exec: 'git' 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
14:05:26.135493 git.c:371   trace: built-in: git 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
fatal: Out of memory, getdelim failed
error: git://github.com/datalad/datalad did not send all necessary objects

14:05:26.138838 run-command.c:350   trace: run_command: 'gc' '--auto'
14:05:26.139131 exec_cmd.c:116  trace: exec: 'git' 'gc' '--auto'
14:05:26.141108 git.c:371   trace: built-in: git 'gc' '--auto'


yhalchen@discovery:/mnt/scratch/yoh$ cat git_trace_good.log
14:05:37.851862 git.c:371   trace: built-in: git 'pull' '--ff-only' 
'origin' 'master'
14:05:37.854250 run-command.c:350   trace: run_command: 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:37.854527 exec_cmd.c:116  trace: exec: 'git' 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:37.856389 

Re: fatal: Out of memory, getdelim failed under NFS mounts

[René Scharfe]

Am 09.08.2017 um 19:39 schrieb Yaroslav Halchenko:
> More context (may be different issue(s)) could be found at
> http://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/
> but currently I am consistently reproducing it while running
> git (1:2.11.0-3 debian stretch build) within debian stretch singularity
> environment [1].
> 
> External system is Centos 6.9, and git 1.7.1 (and installed in modules
> 2.0.4) do not show similar buggy behavior.
> 
> NFS mounted partitions are bind mounted inside the sinularity space and
> when I try to do some git operations, I get that error inconsistently , e.g.
> 
>   yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> master
>   fatal: Out of memory, getdelim failed
>   error: git://github.com/datalad/datalad did not send all necessary 
> objects
> 
>   yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> master
>   fatal: Out of memory, getdelim failed
>   error: git://github.com/datalad/datalad did not send all necessary 
> objects
> 
>   yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> master
>   From git://github.com/datalad/datalad
>* branch  master -> FETCH_HEAD
>   fatal: Out of memory, getdelim failed
> 
> and some times it succeeds.  So it smells that some race condition
> somewhere...?

I doubt the type of file system matters.  The questions are: How much
main memory do you have, what is git trying to cram into it, is there
a way to reduce the memory footprint or do you need to add more RAM?

> any recommendations on how to pin point the "offender"? ;)
Running "GIT_TRACE=1 git pull --ff-only origin master" would be a
good start, I think, to find out which of the different activities
that pull is doing causes the out-of-memory error.

"free" and "ulimit -a" can help you find out how much memory you can
use.

Also: What does "wc -L .git/FETCH_HEAD .git/packed-refs" report?
getdelim() is used mostly to read lines from files like these and in
the admittedly unlikely case that they are *really* long such an
error would be expected.

René

fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

Dear Git gurus,

More context (may be different issue(s)) could be found at 
http://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/
but currently I am consistently reproducing it while running 
git (1:2.11.0-3 debian stretch build) within debian stretch singularity
environment [1].  

External system is Centos 6.9, and git 1.7.1 (and installed in modules
2.0.4) do not show similar buggy behavior.

NFS mounted partitions are bind mounted inside the sinularity space and
when I try to do some git operations, I get that error inconsistently , e.g.

yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
master
fatal: Out of memory, getdelim failed
error: git://github.com/datalad/datalad did not send all necessary 
objects

yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
master
fatal: Out of memory, getdelim failed
error: git://github.com/datalad/datalad did not send all necessary 
objects

yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
master
From git://github.com/datalad/datalad
 * branch  master -> FETCH_HEAD
fatal: Out of memory, getdelim failed

and some times it succeeds.  So it smells that some race condition
somewhere...?

any recommendations on how to pin point the "offender"? ;)  Here is the
trailer of one of the straced calls:

...
[pid 19713] getcwd("/ihome/yhalchen/datalad", 129) = 24
[pid 19713] stat(".git", {st_mode=S_IFDIR|0755, st_size=322, ...}) = 0
[pid 19713] lstat(".git/HEAD", {st_mode=S_IFREG|0644, st_size=41, ...}) = 0
[pid 19713] open(".git/HEAD", O_RDONLY) = 3
[pid 19713] read(3, "39f80454d31cfb691b006302b1f29dee"..., 255) = 41
[pid 19713] read(3, "", 214)= 0
[pid 19713] close(3)= 0
[pid 19713] lstat(".git/commondir", 0x7ffc1a571190) = -1 ENOENT (No such file 
or directory)
[pid 19713] access(".git/objects", X_OK) = 0
[pid 19713] access(".git/refs", X_OK)   = 0
[pid 19713] lstat(".git/commondir", 0x7ffc1a571120) = -1 ENOENT (No such file 
or directory)
[pid 19713] open(".git/config", O_RDONLY) = 3
[pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0
[pid 19713] mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
-1, 0) = 0x2ba7894e7000
[pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257
[pid 19713] read(3, "", 524288) = 0
[pid 19713] close(3)= 0
[pid 19713] munmap(0x2ba7894e7000, 528384) = 0
[pid 19713] stat(".", {st_mode=S_IFDIR|0755, st_size=907, ...}) = 0
[pid 19713] getcwd("/ihome/yhalchen/datalad", 129) = 24
[pid 19713] chdir(".")  = 0
[pid 19713] getcwd("/ihome/yhalchen/datalad", 130) = 24
[pid 19713] lstat("/ihome/yhalchen/datalad", {st_mode=S_IFDIR|0755, 
st_size=907, ...}) = 0
[pid 19713] chdir("/ihome/yhalchen/datalad") = 0
[pid 19713] stat(".git", {st_mode=S_IFDIR|0755, st_size=322, ...}) = 0
[pid 19713] lstat(".git/commondir", 0x7ffc1a571140) = -1 ENOENT (No such file 
or directory)
[pid 19713] access("/etc/gitconfig", R_OK) = -1 ENOENT (No such file or 
directory)
[pid 19713] access("/ihome/yhalchen/.config/git/config", R_OK) = -1 ENOENT (No 
such file or directory)
[pid 19713] access("/ihome/yhalchen/.gitconfig", R_OK) = -1 ENOENT (No such 
file or directory)
[pid 19713] access(".git/config", R_OK) = 0
[pid 19713] open(".git/config", O_RDONLY) = 3
[pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0
[pid 19713] brk(0x2ba78afea000) = 0x2ba78afea000
[pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257
[pid 19713] read(3, "", 524288) = 0
[pid 19713] close(3)= 0
[pid 19713] access("/etc/gitconfig", R_OK) = -1 ENOENT (No such file or 
directory)
[pid 19713] access("/ihome/yhalchen/.config/git/config", R_OK) = -1 ENOENT (No 
such file or directory)
[pid 19713] access("/ihome/yhalchen/.gitconfig", R_OK) = -1 ENOENT (No such 
file or directory)
[pid 19713] access(".git/config", R_OK) = 0
[pid 19713] open(".git/config", O_RDONLY) = 3
[pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0
[pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257
[pid 19713] read(3, "", 524288) = 0
[pid 19713] close(3)= 0
[pid 19713] open(".git/objects/pack", 
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
[pid 19713] fstat(3, {st_mode=S_IFDIR|0755, st_size=270, ...}) = 0
[pid 19713] getdents(3, /* 6 entries */, 524288) = 336
[pid 19713] 
access(".git/objects/pack/pack-fd9e70075570d8ec41f12605852f54f1cb9771a8.keep", 
F_OK) = -1 ENOENT (No such file or directory)
[pid 19713] 
stat(".git/objects/pack/pack-fd9e70075570d8ec41f12605852f54f1cb9771a8.pack", 
{st_mode=S_IFREG|0444, st_size=10019975, ...}) = 0
[pid 19713] 
access(".git/objects/pack/pack-f1fc124e3aa1619d65a6ba56219f84871a762775.keep", 
F_OK) = -1 ENOENT (No such file or directory)
[pid 19713]