Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[René Scharfe]

Am 29.03.2017 um 06:54 schrieb Christian Couder:

On Tue, Mar 28, 2017 at 11:49 PM, Jeff King  wrote:

On Tue, Mar 28, 2017 at 11:15:12PM +0200, Christian Couder wrote:


On Sun, Mar 26, 2017 at 3:43 PM, René Scharfe  wrote:

FreeBSD implements getcwd(3) as a syscall, but falls back to a version
based on readdir(3) if it fails for some reason.  The latter requires
permissions to read and execute path components, while the former does
not.  That means that if our buffer is too small and we're missing
rights we could get EACCES, but we may succeed with a bigger buffer.

Keep retrying if getcwd(3) indicates lack of permissions until our
buffer can fit PATH_MAX bytes, as that's the maximum supported by the
syscall on FreeBSD anyway.  This way we do what we can to be able to
benefit from the syscall, but we also won't loop forever if there is a
real permission issue.


Sorry to be late and maybe I missed something obvious, but the above
and the patch seem complex to me compared with something like:

diff --git a/strbuf.c b/strbuf.c
index ace58e7367..25eadcbedc 100644
--- a/strbuf.c
+++ b/strbuf.c
@@ -441,7 +441,7 @@ int strbuf_readlink(struct strbuf *sb, const char
*path, size_t hint)
 int strbuf_getcwd(struct strbuf *sb)
 {
size_t oldalloc = sb->alloc;
-   size_t guessed_len = 128;
+   size_t guessed_len = PATH_MAX > 128 ? PATH_MAX : 128;

for (;; guessed_len *= 2) {
strbuf_grow(sb, guessed_len);


I think the main reason is just that we do not have to pay the price to
allocate PATH_MAX-sized buffers when they are rarely used.

I doubt it matters all that much in practice, though.


Yeah, I can understand that.


Right, using PATH_MAX as initial size (no ?: operator needed) is the 
easiest fix.  And yes, I wanted to avoid wasting memory just to fix an 
odd special case.


Given that the function isn't called very often the impact is probably 
not that high either way, but increasing memory usage by 3200% on Linux 
just didn't sit right with me -- even though 4KB isn't much in absolute 
terms, of course.



But I also wonder if René's patch relies on PATH_MAX being a multiple
of 128 on FreeBSD and what would happen for a path between 129 and
PATH_MAX if PATH_MAX was something like 254.


We can use any value bigger than zero to initialize guessed_len. 
getcwd() won't have any problems fitting e.g. a path with a length of 
130 bytes into a buffer of 256 bytes in the second round of the loop.


René

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Christian Couder]

On Tue, Mar 28, 2017 at 11:49 PM, Jeff King  wrote:
> On Tue, Mar 28, 2017 at 11:15:12PM +0200, Christian Couder wrote:
>
>> On Sun, Mar 26, 2017 at 3:43 PM, René Scharfe  wrote:
>> > FreeBSD implements getcwd(3) as a syscall, but falls back to a version
>> > based on readdir(3) if it fails for some reason.  The latter requires
>> > permissions to read and execute path components, while the former does
>> > not.  That means that if our buffer is too small and we're missing
>> > rights we could get EACCES, but we may succeed with a bigger buffer.
>> >
>> > Keep retrying if getcwd(3) indicates lack of permissions until our
>> > buffer can fit PATH_MAX bytes, as that's the maximum supported by the
>> > syscall on FreeBSD anyway.  This way we do what we can to be able to
>> > benefit from the syscall, but we also won't loop forever if there is a
>> > real permission issue.
>>
>> Sorry to be late and maybe I missed something obvious, but the above
>> and the patch seem complex to me compared with something like:
>>
>> diff --git a/strbuf.c b/strbuf.c
>> index ace58e7367..25eadcbedc 100644
>> --- a/strbuf.c
>> +++ b/strbuf.c
>> @@ -441,7 +441,7 @@ int strbuf_readlink(struct strbuf *sb, const char
>> *path, size_t hint)
>>  int strbuf_getcwd(struct strbuf *sb)
>>  {
>> size_t oldalloc = sb->alloc;
>> -   size_t guessed_len = 128;
>> +   size_t guessed_len = PATH_MAX > 128 ? PATH_MAX : 128;
>>
>> for (;; guessed_len *= 2) {
>> strbuf_grow(sb, guessed_len);
>
> I think the main reason is just that we do not have to pay the price to
> allocate PATH_MAX-sized buffers when they are rarely used.
>
> I doubt it matters all that much in practice, though.

Yeah, I can understand that.

But I also wonder if René's patch relies on PATH_MAX being a multiple
of 128 on FreeBSD and what would happen for a path between 129 and
PATH_MAX if PATH_MAX was something like 254.

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Jeff King]

On Tue, Mar 28, 2017 at 11:15:12PM +0200, Christian Couder wrote:

> On Sun, Mar 26, 2017 at 3:43 PM, René Scharfe  wrote:
> > FreeBSD implements getcwd(3) as a syscall, but falls back to a version
> > based on readdir(3) if it fails for some reason.  The latter requires
> > permissions to read and execute path components, while the former does
> > not.  That means that if our buffer is too small and we're missing
> > rights we could get EACCES, but we may succeed with a bigger buffer.
> >
> > Keep retrying if getcwd(3) indicates lack of permissions until our
> > buffer can fit PATH_MAX bytes, as that's the maximum supported by the
> > syscall on FreeBSD anyway.  This way we do what we can to be able to
> > benefit from the syscall, but we also won't loop forever if there is a
> > real permission issue.
> 
> Sorry to be late and maybe I missed something obvious, but the above
> and the patch seem complex to me compared with something like:
> 
> diff --git a/strbuf.c b/strbuf.c
> index ace58e7367..25eadcbedc 100644
> --- a/strbuf.c
> +++ b/strbuf.c
> @@ -441,7 +441,7 @@ int strbuf_readlink(struct strbuf *sb, const char
> *path, size_t hint)
>  int strbuf_getcwd(struct strbuf *sb)
>  {
> size_t oldalloc = sb->alloc;
> -   size_t guessed_len = 128;
> +   size_t guessed_len = PATH_MAX > 128 ? PATH_MAX : 128;
> 
> for (;; guessed_len *= 2) {
> strbuf_grow(sb, guessed_len);

I think the main reason is just that we do not have to pay the price to
allocate PATH_MAX-sized buffers when they are rarely used.

I doubt it matters all that much in practice, though.

-Peff

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Christian Couder]

On Tue, Mar 28, 2017 at 11:24 PM, Stefan Beller  wrote:
> On Tue, Mar 28, 2017 at 2:15 PM, Christian Couder
>  wrote:
>> On Sun, Mar 26, 2017 at 3:43 PM, René Scharfe  wrote:
>>> FreeBSD implements getcwd(3) as a syscall, but falls back to a version
>>> based on readdir(3) if it fails for some reason.  The latter requires
>>> permissions to read and execute path components, while the former does
>>> not.  That means that if our buffer is too small and we're missing
>>> rights we could get EACCES, but we may succeed with a bigger buffer.
>>>
>>> Keep retrying if getcwd(3) indicates lack of permissions until our
>>> buffer can fit PATH_MAX bytes, as that's the maximum supported by the
>>> syscall on FreeBSD anyway.  This way we do what we can to be able to
>>> benefit from the syscall, but we also won't loop forever if there is a
>>> real permission issue.
>>
>> Sorry to be late and maybe I missed something obvious, but the above
>> and the patch seem complex to me compared with something like:
>>
>> diff --git a/strbuf.c b/strbuf.c
>> index ace58e7367..25eadcbedc 100644
>> --- a/strbuf.c
>> +++ b/strbuf.c
>> @@ -441,7 +441,7 @@ int strbuf_readlink(struct strbuf *sb, const char
>> *path, size_t hint)
>>  int strbuf_getcwd(struct strbuf *sb)
>>  {
>> size_t oldalloc = sb->alloc;
>> -   size_t guessed_len = 128;
>> +   size_t guessed_len = PATH_MAX > 128 ? PATH_MAX : 128;
>>
>> for (;; guessed_len *= 2) {
>> strbuf_grow(sb, guessed_len);
>
> From f22a76e911 (strbuf: add strbuf_getcwd(), 2014-07-28):
> Because it doesn't use a fixed-size buffer it supports
> arbitrarily long paths, provided the platform's getcwd() does as well.
> At least on Linux and FreeBSD it handles paths longer than PATH_MAX
> just fine.

Well René's patch says:

>>> Keep retrying if getcwd(3) indicates lack of permissions until our
>>> buffer can fit PATH_MAX bytes, as that's the maximum supported by the
>>> syscall on FreeBSD anyway.

So it says that FreeBSD syscall doesn't handle paths longer than PATH_MAX.

> So with your patch, we'd still see the original issue for paths > PATH_MAX
> IIUC.

Also, René's patch just adds:

+   if (errno == EACCES && guessed_len < PATH_MAX)
+   continue;

so if the length of the path is > PATH_MAX, then guessed_len will have
to be > PATH_MAX and then René's patch will do nothing.

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Stefan Beller]

On Tue, Mar 28, 2017 at 2:15 PM, Christian Couder
 wrote:
> On Sun, Mar 26, 2017 at 3:43 PM, René Scharfe  wrote:
>> FreeBSD implements getcwd(3) as a syscall, but falls back to a version
>> based on readdir(3) if it fails for some reason.  The latter requires
>> permissions to read and execute path components, while the former does
>> not.  That means that if our buffer is too small and we're missing
>> rights we could get EACCES, but we may succeed with a bigger buffer.
>>
>> Keep retrying if getcwd(3) indicates lack of permissions until our
>> buffer can fit PATH_MAX bytes, as that's the maximum supported by the
>> syscall on FreeBSD anyway.  This way we do what we can to be able to
>> benefit from the syscall, but we also won't loop forever if there is a
>> real permission issue.
>
> Sorry to be late and maybe I missed something obvious, but the above
> and the patch seem complex to me compared with something like:
>
> diff --git a/strbuf.c b/strbuf.c
> index ace58e7367..25eadcbedc 100644
> --- a/strbuf.c
> +++ b/strbuf.c
> @@ -441,7 +441,7 @@ int strbuf_readlink(struct strbuf *sb, const char
> *path, size_t hint)
>  int strbuf_getcwd(struct strbuf *sb)
>  {
> size_t oldalloc = sb->alloc;
> -   size_t guessed_len = 128;
> +   size_t guessed_len = PATH_MAX > 128 ? PATH_MAX : 128;
>
> for (;; guessed_len *= 2) {
> strbuf_grow(sb, guessed_len);

>From f22a76e911 (strbuf: add strbuf_getcwd(), 2014-07-28):
Because it doesn't use a fixed-size buffer it supports
arbitrarily long paths, provided the platform's getcwd() does as well.
At least on Linux and FreeBSD it handles paths longer than PATH_MAX
just fine.

So with your patch, we'd still see the original issue for paths > PATH_MAX
IIUC.

Thanks,
Stefan

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Christian Couder]

On Sun, Mar 26, 2017 at 3:43 PM, René Scharfe  wrote:
> FreeBSD implements getcwd(3) as a syscall, but falls back to a version
> based on readdir(3) if it fails for some reason.  The latter requires
> permissions to read and execute path components, while the former does
> not.  That means that if our buffer is too small and we're missing
> rights we could get EACCES, but we may succeed with a bigger buffer.
>
> Keep retrying if getcwd(3) indicates lack of permissions until our
> buffer can fit PATH_MAX bytes, as that's the maximum supported by the
> syscall on FreeBSD anyway.  This way we do what we can to be able to
> benefit from the syscall, but we also won't loop forever if there is a
> real permission issue.

Sorry to be late and maybe I missed something obvious, but the above
and the patch seem complex to me compared with something like:

diff --git a/strbuf.c b/strbuf.c
index ace58e7367..25eadcbedc 100644
--- a/strbuf.c
+++ b/strbuf.c
@@ -441,7 +441,7 @@ int strbuf_readlink(struct strbuf *sb, const char
*path, size_t hint)
 int strbuf_getcwd(struct strbuf *sb)
 {
size_t oldalloc = sb->alloc;
-   size_t guessed_len = 128;
+   size_t guessed_len = PATH_MAX > 128 ? PATH_MAX : 128;

for (;; guessed_len *= 2) {
strbuf_grow(sb, guessed_len);

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Junio C Hamano]

Zenobiusz Kunegunda <zenobiusz.kunegu...@interia.pl> writes:

> Od: "Junio C Hamano" <gits...@pobox.com>
> Do: "René Scharfe" <l@web.de>;
> Wysłane: 2:40 Poniedziałek 2017-03-27
> Temat: Re: [PATCH] strbuf: support long paths w/o read rights in 
> strbuf_getcwd() on FreeBSD
>
>> 
>> Nicely analysed and fixed (or is the right word "worked around"?)
>> 
>> Thanks, will queue.
>
> Is this patch going to be included in next git version ( or sooner ) by any 
> chance?

Absolutely.  Thanks for your initial report and sticking with us
during the session to identify the root cause that led to this
solution.

Again, René, thanks for your superb analysis and solution.

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Zenobiusz Kunegunda]

Od: "Junio C Hamano" <gits...@pobox.com>
Do: "René Scharfe" <l@web.de>;
Wysłane: 2:40 Poniedziałek 2017-03-27
Temat: Re: [PATCH] strbuf: support long paths w/o read rights in 
strbuf_getcwd() on FreeBSD

> 
> Nicely analysed and fixed (or is the right word "worked around"?)
> 
> Thanks, will queue.
> 


Is this patch going to be included in next git version ( or sooner ) by any 
chance?

Thank you, everyone,  for your attention to the problem.

Re: [PATCH] strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD

[Junio C Hamano]

René Scharfe  writes:

> FreeBSD implements getcwd(3) as a syscall, but falls back to a version
> based on readdir(3) if it fails for some reason.  The latter requires
> permissions to read and execute path components, while the former does
> not.  That means that if our buffer is too small and we're missing
> rights we could get EACCES, but we may succeed with a bigger buffer.

WOW.  Just WOW.  Looking at the debugging exchange from the
sideline, I didn't expect this end result.

> Keep retrying if getcwd(3) indicates lack of permissions until our
> buffer can fit PATH_MAX bytes, as that's the maximum supported by the
> syscall on FreeBSD anyway.  This way we do what we can to be able to
> benefit from the syscall, but we also won't loop forever if there is a
> real permission issue.
>
> This fixes a regression introduced with 7333ed17 (setup: convert
> setup_git_directory_gently_1 et al. to strbuf, 2014-07-28) for paths
> longer than 127 bytes with components that miss read or execute
> permissions (e.g. 0711 on /home for privacy reasons); we used a fixed
> PATH_MAX-sized buffer before.
>
> Reported-by: Zenobiusz Kunegunda 
> Signed-off-by: Rene Scharfe 
> ---


Nicely analysed and fixed (or is the right word "worked around"?)

Thanks, will queue.