Bug#807836: [Pkg-zsh-devel] Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Thilo Six]

Hello

i thought i had sent this here already to bts. Turn out i missed it.
So here it goes.


Thilo Six schrieb/wrote:
> 
> just a short addition, see below
> 
 May i ask then what is a real world use case for the unlimit builtin then?
 (Just to make it clear, i am not ironic here, just interested trying to
 understand it.)
>>>
>>> You might as well ask what's the purpose of having separate soft and
>>> hard limits.
>>>
>>> The hard limit is the value that even malicious or compromised user accounts
>>> may not exceed.
>>>
>>> I do not know what's the historical reason for having a separate soft
>>> limit.  However, I can imagine a multiuser system with a social
>>> expectation that if you run into the soft limit you reschedule your
>>> resource-heavy job to off-peak hours.  Or on a desktop system, you might
>>> selectively increase the limit for specific resource-hungry
>>> applications, giving each application a limit of, say, 5% above its
>>> normal usage.
>>
>> Now we are back were this bug started.
>> I do not argue soft vs. hard limit or their existence at all. They are 
>> useful.
>> I do not even argue that temporarily lifting current limits upwards is 
>> useful.
>>
>> I do argue (and that is what has caused this bug) that setting a limit that 
>> far
>> beyond anything capable on this current system that it is not even technical
>> able to handle that size of such a limit is useful.
>> And that is just what unlimit does (at least that is what i gather from this
>> bugs history).
>>
>> After unlimit has been run the max input size is set to a that large integer
>> that i would need to go and buy a rather professional SAN System to back that
>> setting up with actually s.th. capable for it.
>> And just that is what i simply do not understand, as that behaviour of 
>> unlimit
>> is the opposite of being useful in real world.

I just verified upstream git also has this:

,[  ./StartupFiles/zshrc  ]
# Use hard limits, except for a smaller stack and no core dumps
unlimit
`---

This very example has lead me to the impression that setting this would actually
be ok. Which turned out not. So i suggest to at least comment that out and add a
pointer to this discussion here. Maybe this will prevent some other naive user
from falling into the same pit as happend to me.


>> Now if unlimit would instead evaluate the maximum physical possible size for
>> that limit and activate that, i would say nice.
>> As in your example that would be as pushing the super-turbo-charger button 
>> for
>> current session. And actually that is what i had expected when using it.
>>
>> question:
>> Something i was not be able to 100% verify up to now, but i guess from what i
>> read so far that input size on shell prompt relates to stack size in terms of
>> rlimit?
>>
>> After all if nothing else this bug has served my self for learning quite a 
>> bit
>> so far. At least i do count that as "worth it".
>>
>> Again thanks for your input. It is appreciated.
>>
>>
>>
>> kind regards,
>>
>>  Thilo
>>
> 

Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Thilo Six]

Hello Daniel,

Daniel Shahaf schrieb/wrote:

>> Now if unlimit would instead evaluate the maximum physical possible size for
>> that limit and activate that, i would say nice.
> 
> Managing system resources is the OS's job, not zsh's.  I suggest you
> look into configuring your OS to set the hard limit as a function of the
> available hardware.  Then, calling 'unlimit' in zsh will behave as you
> expect (without any code changes to zsh or to your zsh script).

Ahh that sounds like the one missing puzzle piece that would glue it all
together. I will dig this up.

>> question:
>> Something i was not be able to 100% verify up to now, but i guess from what i
>> read so far that input size on shell prompt relates to stack size in terms of
>> rlimit?
>>
> 
> The stack contains function call frames.  The input to the shell would
> be allocated on the heap, not on the stack.

I am not sure i will be able to make any sense for my self out of this. But i
will try. Currently i am running a bit out of time. So this will take a larger
timeframe then the past few days.

> Hence, it would be affected
> by the memory allocation limits, not by the stack size limits.

Thanks for the hint. This will help.

Again Thanks Daniel for taking the time to shed some light on this.

@Axel:
At this point i tend to agree that this bug here is invalid. As the discussion
here has culminated some "behind the scenes" it might still be good a idea to
add a pointer towards here for others. Here is hope that someone else will find
it useful.
What is the correct status to set on this issue here? Closed or wont fix?
I suspect rather the former then the latter?

Thanks!


kind regards,

 Thilo

Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Daniel Shahaf]

Thilo Six wrote on Sat, Jan 02, 2016 at 20:28:39 +0100:
> I do argue (and that is what has caused this bug) that setting a limit that 
> far
> beyond anything capable on this current system that it is not even technical
> able to handle that size of such a limit is useful.
> And that is just what unlimit does (at least that is what i gather from this
> bugs history).

> Now if unlimit would instead evaluate the maximum physical possible size for
> that limit and activate that, i would say nice.

Managing system resources is the OS's job, not zsh's.  I suggest you
look into configuring your OS to set the hard limit as a function of the
available hardware.  Then, calling 'unlimit' in zsh will behave as you
expect (without any code changes to zsh or to your zsh script).

> question:
> Something i was not be able to 100% verify up to now, but i guess from what i
> read so far that input size on shell prompt relates to stack size in terms of
> rlimit?
> 

The stack contains function call frames.  The input to the shell would
be allocated on the heap, not on the stack.  Hence, it would be affected
by the memory allocation limits, not by the stack size limits.

Cheers,

Daniel

Bug#807836: [Pkg-zsh-devel] Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Daniel Shahaf]

Control: retitle -1 zshbuiltins: Clarify the meaning of "hard limit" in the 
documentation of 'unlimit'
Control: severity -1 wishlist

Thilo Six wrote on Thu, Dec 31, 2015 at 12:19:19 +0100:
> Daniel Shahaf schrieb/wrote:
> > Thilo Six wrote on Wed, Dec 30, 2015 at 17:12:17 +0100:
> >> So as i said above, the definition of "hard limit" should be explicit in 
> >> the man
> >> page, e.g.:
> >> The resource limit for each resource is set to the hard limit, that is ...
> > 
> > Maybe.  I honestly don't have an opinion — I don't find the man page
> > ambiguous in the first place ☺.
> 
> May i ask then what is a real world use case for the unlimit builtin then?
> (Just to make it clear, i am not ironic here, just interested trying to
> understand it.)

You might as well ask what's the purpose of having separate soft and
hard limits.

The hard limit is the value that even malicious or compromised user accounts
may not exceed.

I do not know what's the historical reason for having a separate soft
limit.  However, I can imagine a multiuser system with a social
expectation that if you run into the soft limit you reschedule your
resource-heavy job to off-peak hours.  Or on a desktop system, you might
selectively increase the limit for specific resource-hungry
applications, giving each application a limit of, say, 5% above its
normal usage.

Cheers,

Daniel

Bug#807836: [Pkg-zsh-devel] Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Thilo Six]

Hello Daniel,

Daniel Shahaf schrieb/wrote:

>> May i ask then what is a real world use case for the unlimit builtin then?
>> (Just to make it clear, i am not ironic here, just interested trying to
>> understand it.)
> 
> You might as well ask what's the purpose of having separate soft and
> hard limits.
> 
> The hard limit is the value that even malicious or compromised user accounts
> may not exceed.
> 
> I do not know what's the historical reason for having a separate soft
> limit.  However, I can imagine a multiuser system with a social
> expectation that if you run into the soft limit you reschedule your
> resource-heavy job to off-peak hours.  Or on a desktop system, you might
> selectively increase the limit for specific resource-hungry
> applications, giving each application a limit of, say, 5% above its
> normal usage.

Now we are back were this bug started.
I do not argue soft vs. hard limit or their existence at all. They are useful.
I do not even argue that temporarily lifting current limits upwards is useful.

I do argue (and that is what has caused this bug) that setting a limit that far
beyond anything capable on this current system that it is not even technical
able to handle that size of such a limit is useful.
And that is just what unlimit does (at least that is what i gather from this
bugs history).

After unlimit has been run the max input size is set to a that large integer
that i would need to go and buy a rather professional SAN System to back that
setting up with actually s.th. capable for it.
And just that is what i simply do not understand, as that behaviour of unlimit
is the opposite of being useful in real world.

Now if unlimit would instead evaluate the maximum physical possible size for
that limit and activate that, i would say nice.
As in your example that would be as pushing the super-turbo-charger button for
current session. And actually that is what i had expected when using it.

question:
Something i was not be able to 100% verify up to now, but i guess from what i
read so far that input size on shell prompt relates to stack size in terms of
rlimit?

After all if nothing else this bug has served my self for learning quite a bit
so far. At least i do count that as "worth it".

Again thanks for your input. It is appreciated.



kind regards,

 Thilo

Bug#807836: [Pkg-zsh-devel] Bug#807836: Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Thilo Six]

Daniel Shahaf schrieb/wrote:
> Thilo Six wrote on Wed, Dec 30, 2015 at 17:12:17 +0100:
>> The manpage says (as mentioned earlier):
>>   unlimit [ -hs ] resource ...
>>   The resource limit for each resource is set to the  hard  
>> limit.
>>
>> Again "hard limit" by its own is as ambiguous as "unlimited".
>>
> 
> The intended meaning of "hard limit" in this context is "the value of
> the 'rlim_max' member of 'struct rlimit'".  The getrlimit(2) man page
> (and its POSIX spec) use that term with the same meaning.

Up front i have to say i don't speak C.
That being said i digged up the pointers you gave and i admit the unlimit
behaves acordingly to the specs.
(Side note: Actually that is nothing i ever argued against.)

,[  getrlimit(2)  ]
A  privileged  process  (under  Linux:  one  with  the CAP_SYS_RESOURCE
capability) may make arbitrary changes to either limit value.
`---

So unlimited in this context indeed means ∞ and it may be changed arbitrary.

hmmm

>> So as i said above, the definition of "hard limit" should be explicit in the 
>> man
>> page, e.g.:
>> The resource limit for each resource is set to the hard limit, that is ...
> 
> Maybe.  I honestly don't have an opinion — I don't find the man page
> ambiguous in the first place ☺.

May i ask then what is a real world use case for the unlimit builtin then?
(Just to make it clear, i am not ironic here, just interested trying to
understand it.)

Thank you for your answer.



kind regards,

 Thilo

Bug#807836: [Pkg-zsh-devel] Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Thilo Six]

Thilo Six schrieb/wrote:
--  --

>> $ ulimit -v -m
>> virtual memory  (kbytes, -v) unlimited
>> max memory size (kbytes, -m) unlimited
>>
>> I'm sure even with swap, there's no unlimited virtual memory available
>> on my machine.
> 
> 
> That is my main point. I think the "real problem" here is the definition of a
> "hard limit". My definition of a "hard limit" is either the maximum physical
> capability of the hardware being used, or a subset of that when a admin sets a
> hard limit via e.g. /etc/security/limits.conf .
> ...and not some random value without any meaning.

I thought it would be helpful to be more wordy about what i mean.


unlimited by its own is ambiguous.

unlimited can mean ∞
but IMHO an other valid meaning of unlimited is:
"as much, as there is without restrictions"

And that is what i am talking about.

The manpage says (as mentioned earlier):
  unlimit [ -hs ] resource ...
  The resource limit for each resource is set to the  hard  limit.

Again "hard limit" by its own is as ambiguous as "unlimited".

So as i said above, the definition of "hard limit" should be explicit in the man
page, e.g.:
The resource limit for each resource is set to the hard limit, that is ...

I want to add, i do not force Zsh maintainers to choose either of the meanings
again i just ask Zsh maintainers kindly to clearly document which one they have
choosen.

I hope my point which lead to this bug is now more easier to follow.


kind regards,

 Thilo

Bug#807836: [Pkg-zsh-devel] Bug#807836: Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Daniel Shahaf]

Thilo Six wrote on Wed, Dec 30, 2015 at 17:12:17 +0100:
> The manpage says (as mentioned earlier):
>   unlimit [ -hs ] resource ...
>   The resource limit for each resource is set to the  hard  limit.
> 
> Again "hard limit" by its own is as ambiguous as "unlimited".
> 

The intended meaning of "hard limit" in this context is "the value of
the 'rlim_max' member of 'struct rlimit'".  The getrlimit(2) man page
(and its POSIX spec) use that term with the same meaning.

> So as i said above, the definition of "hard limit" should be explicit in the 
> man
> page, e.g.:
> The resource limit for each resource is set to the hard limit, that is ...

Maybe.  I honestly don't have an opinion — I don't find the man page
ambiguous in the first place ☺.

Cheers,

Daniel

Bug#807836: [Pkg-zsh-devel] Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Axel Beckert]

Hi Thilo,

Thilo Six wrote:
> in https://bugs.debian.org/807836 it was discovered that the Zsh builtin
> "unlimit" has potential unforeseen side-effects.
> 
> An easy to reproduce case for the problem is documented in
> https://bugs.debian.org/807836#30

That example is IMHO a wrong usage of awk then. If the consuming
script isn't capable of parsing scientific notation but uses a tool
which produces scientific notation if needed, that's for sure not
zsh's fault.

> The problem here is that unlimit sets the maximum input size to s.th. that is 
> by
> far out of the feasibility of the hardware being used.

So what. Bash does similar things:

$ ulimit -v -m
virtual memory  (kbytes, -v) unlimited
max memory size (kbytes, -m) unlimited

I'm sure even with swap, there's no unlimited virtual memory available
on my machine.

> This results in breakage of programs that reuse that capability for their own
> processing.

So while it could be a bug that such big values are output despite
they should be smaller, as you say, it doesn't need to be a bug.

> I contact you to ask if that Zsh behaviour of "unlimit" is actually by design 
> or
> if this is a bug which should be fixed.

We'll check with upstream.

> If the former i ask if the documentation of the "unlimit" builtin can be
> extended to inform users that the usage of "unlimit" has potential unforeseen
> side-effects which might break valid code.

I'm sorry, but if code is broken by too big integer values or
unexpected by used scientific notation, it's not valid code but broken
code.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#807836: [Pkg-zsh-devel] Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Thilo Six]

Hello Axel,

Axel Beckert schrieb/wrote:

>> in https://bugs.debian.org/807836 it was discovered that the Zsh builtin
>> "unlimit" has potential unforeseen side-effects.
>>
>> An easy to reproduce case for the problem is documented in
>> https://bugs.debian.org/807836#30
> 
> That example is IMHO a wrong usage of awk then. If the consuming
> script isn't capable of parsing scientific notation but uses a tool
> which produces scientific notation if needed, that's for sure not
> zsh's fault.
> 
>> The problem here is that unlimit sets the maximum input size to s.th. that 
>> is by
>> far out of the feasibility of the hardware being used.
> 
> So what. Bash does similar things:
> 
> $ ulimit -v -m
> virtual memory  (kbytes, -v) unlimited
> max memory size (kbytes, -m) unlimited
> 
> I'm sure even with swap, there's no unlimited virtual memory available
> on my machine.


That is my main point. I think the "real problem" here is the definition of a
"hard limit". My definition of a "hard limit" is either the maximum physical
capability of the hardware being used, or a subset of that when a admin sets a
hard limit via e.g. /etc/security/limits.conf .
...and not some random value without any meaning.

Obviously YMMV

So i do not blame anyone, i just ask for clarification on:
i How is unlimit designed to work.
   ii Improved its documentation to match that design, so that potentially users
  of unlimit have a chance to reliable decide whether unlimt is for them.

Hint:
/usr/share/doc/zsh-common/examples/zshrc.gz
contains "unlimit" and thats iirc is the place where i got it from. Nor from
that example file nor from manpage one can have a glimpse of the real impact
that unlimit has for runtime.

Again i do not blame anyone and i certainly will not jump into a potential flame
war over "how to write good code".

It is just that i had fallen into a trap which could have been avoided with more
information about the consequences of unlimit.


> I'm sorry, but if code is broken by too big integer values or
> unexpected by used scientific notation, it's not valid code but broken
> code.

I second that. But that is an other bug.



kind regards,

 Thilo

Bug#807836: builtin unlimit leads to "xargs: invalid number for -s"

[Thilo Six]

Hello Zsh maintainers,


in https://bugs.debian.org/807836 it was discovered that the Zsh builtin
"unlimit" has potential unforeseen side-effects.

An easy to reproduce case for the problem is documented in
https://bugs.debian.org/807836#30

The problem here is that unlimit sets the maximum input size to s.th. that is by
far out of the feasibility of the hardware being used.
This results in breakage of programs that reuse that capability for their own
processing.

I contact you to ask if that Zsh behaviour of "unlimit" is actually by design or
if this is a bug which should be fixed.
If the former i ask if the documentation of the "unlimit" builtin can be
extended to inform users that the usage of "unlimit" has potential unforeseen
side-effects which might break valid code.

Thank you in advance.


kind regards,

 Thilo