Pádraig Brady <[email protected]> writes:
> On 11/10/2025 19:50, Collin Funk wrote:
>> Collin Funk <[email protected]> writes:
>>
>>> $ timeout 10 sh -c \
>>> "(ulimit -v $s && uutils base64 < /dev/zero >out 2>err)"
>>> $ cat err && wc -c out
>>> base64: out of memory
>>> 0 out
>> In case anyone is looking for work to do, there are some programs in
>> GNU
>> Coreutils who have this issue. For example, 'basenc --base58' requires
>> that the file fits into memory. It would be nice to fix those where
>> possible.
>> The 'basenc --base58' case, if I remember correctly, is because of
>> the
>> gmp functions we use. If we could fix it without sacrificing the
>> speedups from gmp, that would be great.
>> It also would be a good idea to add bounded-memory.sh tests to other
>> programs.
>
> I definitely would not have added a non streamable base58 implementation
> if a streamable one was supported. Unfortunately the base conversion
> operation used is not streamable.
>
> For example, see the completely different output below for very similar
> inputs:
>
> $ printf '|%*s|' 100 | basenc
> --base582cypwC9WHm46BUbDVHDDtpoEE8HyXVnVw3ytMvBLww1vipUBejw9HGLm3yRvvJstDJqfW11X7nyp
> xsuaWQZgUD4KWXbfomaCWbqNnoejTfzhJqVGbTFU9iHzWsfdy6eZXH3pcT2a9G2w
>
> $ printf '|%*s|' 101 | basenc --base58
> 89opaCCYGySdsEn27pYwujy3NwBvVc3phaA5hKif3TR67aWz1aVyYniDS852zHub3Kjnw13Hvywp
> hdh4to1FD6DehL42HRjXoekXBAuYQj7euj1mq4rrqTFzrrdYAbvnYdqT9TswHTEZR
Thanks for the explanation.
> It's not a huge problem for base58 anyway because it's designed for
> small amounts of data where unambiguous transcription is important.
Yep, that is what I figured from your description of what base58 was
used for when you added. I think all the programs where streamable input
is important, it is done. For example, 'cksum' is obviously important.
It is still a nicety for all programs, in my opinion.
I think 'ptx' is also impossible, or not worth the great effort, to make
streamable, since we have to keep track of all occurances of keywords in
the text:
$ (ulimit -v 1000000 && ptx < /dev/zero)
ptx: -: Cannot allocate memory
Collin
