Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-19 Thread Steffen Prohaska
On Aug 19, 2014, at 9:53 AM, Jeff King wrote: >>> For files >2GB on a 32-bit system (e.g. msysgit), filtering with the >>> previous code always failed. Now it works. I created the patch to >>> change git from 'fundamentally doesn't handle this' to 'works as >>> expected'. >> >> I deal with si

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-19 Thread Jeff King
On Sun, Aug 17, 2014 at 06:33:54PM +0700, Duy Nguyen wrote: > > For files >2GB on a 32-bit system (e.g. msysgit), filtering with the > > previous code always failed. Now it works. I created the patch to > > change git from 'fundamentally doesn't handle this' to 'works as > > expected'. > > I de

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-17 Thread Duy Nguyen
On Sun, Aug 17, 2014 at 5:25 PM, Steffen Prohaska wrote: > > On Aug 17, 2014, at 9:27 AM, Jeff King wrote: > >> On Sat, Aug 16, 2014 at 06:26:08PM +0200, Steffen Prohaska wrote: >> Is the 15MB limit supposed to be imposed somewhere or is it just a guide of how much memory we expect Git

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-17 Thread Andreas Schwab
Steffen Prohaska writes: > On Aug 16, 2014, at 7:00 PM, Andreas Schwab wrote: > >> Steffen Prohaska writes: >> >>> The test should confirm that the the file that is added is not mmapped to >>> memory. >> >> RSS doesn't tell you that. You can mmap a big file without RSS getting >> bigger. > >

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-17 Thread Steffen Prohaska
On Aug 17, 2014, at 9:27 AM, Jeff King wrote: > On Sat, Aug 16, 2014 at 06:26:08PM +0200, Steffen Prohaska wrote: > >>> Is the 15MB limit supposed to be imposed somewhere or is it just a guide >>> of how much memory we expect Git to use in this scenario? >> >> The test should confirm that the

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-17 Thread Steffen Prohaska
On Aug 16, 2014, at 7:00 PM, Andreas Schwab wrote: > Steffen Prohaska writes: > >> The test should confirm that the the file that is added is not mmapped to >> memory. > > RSS doesn't tell you that. You can mmap a big file without RSS getting > bigger. All data are accessed after mapping, s

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-17 Thread Jeff King
On Sat, Aug 16, 2014 at 06:26:08PM +0200, Steffen Prohaska wrote: > > Is the 15MB limit supposed to be imposed somewhere or is it just a guide > > of how much memory we expect Git to use in this scenario? > > The test should confirm that the the file that is added is not mmapped > to memory. The

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-16 Thread Andreas Schwab
Steffen Prohaska writes: > The test should confirm that the the file that is added is not mmapped to > memory. RSS doesn't tell you that. You can mmap a big file without RSS getting bigger. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-16 Thread Steffen Prohaska
On Aug 16, 2014, at 12:27 PM, John Keeping wrote: >> +test_expect_success HAVE_MAX_MEM_USAGE \ >> +'filtering large input to small output should use little memory' ' >> +git config filter.devnull.clean "cat >/dev/null" && >> +git config filter.devnull.required true && >> +for i in $(

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-16 Thread John Keeping
On Wed, Aug 06, 2014 at 07:32:14AM +0200, Steffen Prohaska wrote: [...] > The expectation on the process size is tested using /usr/bin/time. An > alternative would have been tcsh, which could be used to print memory > information as follows: > > tcsh -c 'set time=(0 "%M"); ' > > Although the

[PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

2014-08-05 Thread Steffen Prohaska
The data is streamed to the filter process anyway. Better avoid mapping the file if possible. This is especially useful if a clean filter reduces the size, for example if it computes a sha1 for binary data, like git media. The file size that the previous implementation could handle was limited b