Re: swap-related problems

1999-04-17 Thread Matthew Thyer
There is obviously a problem when all swap is exhausted.

The only solution is to allow the additional memory *use* to succeed AND
to warn the sysadmin that ALL virtual memory has been exhausted.

The only way to do this is to be able to allocate extra virtual memory.
I'd vote for a system that would create swap files in the largest mounted
read/write filesystem of type UFS or in the filesystem of my choice.

There would be a systctl to set the limits on how much space to allocate.
Possibly a setting for size and number of emergency swap files.

When the time comes for killing processes, you should be able to specify
that certain processes (by name) are precious and that processes owned
by particular users and/or particular login classes are in the last resort
or first resort for killing.

I dont think it's worth trying to signal with a different signal because
only FreeBSD specific programs will know what to do when signalled in such
a manner.  I suppose you could signal prior to killing as another layer to
my dream system.

Warner Losh wrote:
 
 In message 199904142340.taa96...@misha.cisco.com Mikhail Teterin writes:
 : Then, one can write a safe malloc, which will install the signal
 : handler, and touch every page in the the memory referenced by the
 : to-be-returned pointer. If the signal handler is invoked in the
 : progress, the to-be-returned memory must be returned back to the
 : system and NULL should be returned to the caller.
 
 This won't work all the time.  FreeBSD overcommits swap space and you
 may get a SIGKILL even if you've touched all the pages.  FreeBSD kills
 processes when swap space runs out.
 
 : However, my (in)ability to propose anything remotely sensible does
 : not change the facts established in this painful thread. That our
 : malloc does not conform to standards (for whatever reasons), and
 : that something should be done about it. That something must start
 : with documenting the flaw...
 
 The behavior is documented:
  The malloc() and calloc() functions return a pointer to the allocated
  memory if successful; otherwise a NULL pointer is returned.
 
 What the system does when it has resource shortages is beyond the
 scope of the ANSI-C standard, so I don't see why you say that
 FreeBSD's malloc isn't standard conforming.
 
 Warner
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message

-- 
/===\
| Work: matthew.th...@dsto.defence.gov.au | Home: thy...@camtech.net.au |
\===/
If it is true that our Universe has a zero net value for all conserved
quantities, then it may simply be a fluctuation of the vacuum of some
larger space in which our Universe is imbedded. In answer to the
question of why it happened, I offer the modest proposal that our
Universe is simply one of those things which happen from time to time.
 E. P. Tryon   from Nature Vol.246 Dec.14, 1973


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-17 Thread Matthew Thyer
Replying to myself...

You'd have to be able to specify the absolute maximum memory use for
a process to ensure you'd still kill run-aways (These would go first!
regardless of the other rules maybe).

Matthew Thyer wrote:
 
 There is obviously a problem when all swap is exhausted.
 
 The only solution is to allow the additional memory *use* to succeed AND
 to warn the sysadmin that ALL virtual memory has been exhausted.
 

-- 
/===\
| Work: matthew.th...@dsto.defence.gov.au | Home: thy...@camtech.net.au |
\===/
If it is true that our Universe has a zero net value for all conserved
quantities, then it may simply be a fluctuation of the vacuum of some
larger space in which our Universe is imbedded. In answer to the
question of why it happened, I offer the modest proposal that our
Universe is simply one of those things which happen from time to time.
 E. P. Tryon   from Nature Vol.246 Dec.14, 1973


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-16 Thread Snob Art Genre
On Thu, 15 Apr 1999, Don Lewis wrote:

 SunOS 4 doesn't do memory overcommit. 

I get the impression from Vahalia's _UNIX Internals_ that this is true
of SunOS 5 as well.

 This could be shaved down a bit if SunOS didn't require
 (swap  total VM) instead of (swap + RAM  total VM).

Again, from my readings it seems that Solaris has the latter policy.
Does anyone know if this is true?  This would seem to put paid to the
idea that overcommit is the only way to go.


 Ben

You have your mind on computers, it seems. 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-16 Thread Poul-Henning Kamp
In message pine.gso.4.10.9904160842580.12675-100...@echonyc.com, Snob Art Gen
re writes:
On Thu, 15 Apr 1999, Don Lewis wrote:

 SunOS 4 doesn't do memory overcommit. 

I get the impression from Vahalia's _UNIX Internals_ that this is true
of SunOS 5 as well.

I know from experience that solaris 2.4 overcommits.

--
Poul-Henning Kamp FreeBSD coreteam member
p...@freebsd.org   Real hackers run -current on their laptop.
FreeBSD -- It will take a long time before progress goes too far!


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Bruce Evans
 Sorry FreeBSD doesn't support resource reservation for memory.

mlock()? BTW, why can't anyone explain why the old behavior WAS THAT
the process would get NULL? Now it's this...

Implementation quirks (bugs) and races.

brk(2) used to fail when memory (real+swap) is running short, instead
of waiting for the pagers to free some memory (it may be possible to
free almost as much memory as there is real memory, e.g., by forgetting
that pages backed by vnodes are in core).  This tended to cause innocent
processes to exit when malloc() failed.  However, in simple tests where
only one memory hog process is calling malloc(), it only affects the
guilty process.

Bruce


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Daniel C. Sobral
Anthony Kimball wrote:
 
 Quoth Daniel C. Sobral on Thu, 15 April:
 :
 : As soon as this application grabs all the memory, the next thing to
 : request more memory (not a malloc!) will cause the application to be
 : killed (largest application criteria). Change the criteria, and
 : *another* application gets killed. Any application getting killed is
 : a problem, you'll find very fast.
 
 Let's not try to solve the unsolvable problem, just the solvable part.  If
 you can't tolerate anything getting killed, you can't use an
 overcommit system, or you must control allocation by manual policy.

What would you have killed? Why not run it in first place? Well, if
you did not run it, it's memory would be taken by the memory-grabber
process, and something else would get killed. So, you need something
killable running just so the memory-grabber will stop at some point
where there is still enough memory free to do the minimum management
necessary (things like cron, inetd -- forget about X), *after* the
killable processes get killed.

Does that sound like taking too many turns? It is. You would have to
measure how much you need, and then have a set of processes just for
the sake of getting killed.

So, let me asking to consider something. Why not setting a datasize
memory limit for the memory-grabbing process instead? Set it to
almost all memory, leaving just enough for minimum usability. Well,
surprisingly (not!), FreeBSD supports that.

In other words, FreeBSD already has a real, working solution. It's
just that it seems not to sink in the head of some people that what
they are asking for doesn't work(*), and what they have been
discarding as a solution does work.

(*) Yeah, a full pre-allocate solution would work too. I just don't
see a point in using that solution since the memory-grabbing program
would have a lot less memory to grab.

--
Daniel C. Sobral(8-DCS)
d...@newsguy.com
d...@freebsd.org

Well, Windows works, using a loose definition of 'works'...




To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Daniel C. Sobral
Anthony Kimball wrote:
 
 Quoth Poul-Henning Kamp on Wed, 14 April:
 :
 : 1. Demonstrate the need.
 
 Well, it's only needed if you want to be able to reliably execute ANSI
 C code according to spec.  I personally don't care.  I'd be surprised
 if core didn't though.  I would suspect that it would be deemed worthy
 of someone's p2 queue, at least.

FreeBSD executes ANSI C code reliably according to spec, as long as
there is enough memory in the system. If there isn't, FreeBSD
doesn't run reliably, much less executes any kind of code reliably.

--
Daniel C. Sobral(8-DCS)
d...@newsguy.com
d...@freebsd.org

Well, Windows works, using a loose definition of 'works'...




To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Daniel C. Sobral
Brian Feldman wrote:
 
 mlock()? BTW, why can't anyone explain why the old behavior WAS THAT
 the process would get NULL? Now it's this...

Maybe the login class was changed, and now has no limits, or a high
enough limit that the resources are exhausted.

--
Daniel C. Sobral(8-DCS)
d...@newsguy.com
d...@freebsd.org

Well, Windows works, using a loose definition of 'works'...



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Tony Finch
Jim Bloom bl...@acm.org wrote:

A signal handler is not guaranteed to work.  It must be written such that it
does not require a new page of memory.  Some possible problems here are the
stack growing, writing on a new page in the data segment, etc.

man sigaltstack

Tony.
-- 
f.a.n.finch   d...@dotat.at   f...@demon.net
Arthur: Oh, that sounds better, have you worked out the controls?
Ford:   No, we just stopped playing with them.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Anthony Kimball
Quoth Daniel C. Sobral on Thu, 15 April:
: 
: FreeBSD executes ANSI C code reliably according to spec, as long as
: there is enough memory in the system. If there isn't, FreeBSD
: doesn't run reliably, much less executes any kind of code reliably.
: 
I understand this.  I concieve the purpose of this thread as being
a clear determination of the best known way to allow correctly 
written code to run reliably when, for reasons perhaps out of the
control of the application, memory becomes overcommitted.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Matthew Dillon

( Sitting in the bleachers )

Ra! Ra! Ra!

-Matt



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Mike Smith
 Jim Bloom bl...@acm.org wrote:
 
 A signal handler is not guaranteed to work.  It must be written such that it
 does not require a new page of memory.  Some possible problems here are the
 stack growing, writing on a new page in the data segment, etc.
 
 man sigaltstack

That doesn't help; there is no guarantee that your stack or your 
alternate stack have been mapped.

-- 
\\  Sometimes you're ahead,   \\  Mike Smith
\\  sometimes you're behind.  \\  m...@smith.net.au
\\  The race is long, and in the  \\  msm...@freebsd.org
\\  end it's only with yourself.  \\  msm...@cdrom.com




To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Tony Finch
Mike Smith writes:
  Jim Bloom bl...@acm.org wrote:
  
  A signal handler is not guaranteed to work.  It must be written such that 
  it
  does not require a new page of memory.  Some possible problems here are the
  stack growing, writing on a new page in the data segment, etc.
  
  man sigaltstack
 
 That doesn't help; there is no guarantee that your stack or your 
 alternate stack have been mapped.

I assumed that the alternate stack had been forcibly allocated by
touching the pages; I suppose things go pear-shaped if they have been
swapped out and there's no space to swap out other pages so that the
stack can be swapped in. Bummer.

Tony.
-- 
f.a.n.finch  d...@dotat.at  f...@demon.net


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Mikhail Teterin
Mike Smith once wrote:

  A signal handler is not guaranteed to work. It must be written
  such that it does not require a new page of memory. Some possible
  problems here are the stack growing, writing on a new page in the
  data segment, etc.
  
  man sigaltstack
 
 That doesn't help; there is no guarantee that your stack or your
 alternate stack have been mapped.

Can I ensure the room availability for the signal handler by explicitly
calling it once, _before_ doing the dangerous operation of touching every
page of the freshly allocated memory?

The thing itself (in the case of earlier described `safe malloc')
will only change a value of a static variable and return, indicating
to the malloc it should:

. stop the touching
. give the memory back to system
. return NULL to the caller.

This assumes, of course, that it is the process doing the malloc
that is sent the signal, AND, that the signal is catchable. The
kernel can try to remember the processes the signal went to, and
send it a SIGKILL if it happens again withing a certain time -- to
avoid the deadlock. Pretty horrible? Feasible?

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: RE: swap-related problems

1999-04-15 Thread Matthew Dillon
Just a quick note... I've heard 'vfork' and 'fork' mentioned twice now.

I think there is some confusion.  fork() does not really eat that
much memory if all you are doing is fork/exec.  The only difference
between fork and vfork from a resource point of view is around 32 KBytes
of memory and a little extra cpu, and even the memory is recovered 
when you do the exec.

Semantically, they are quite different - vfork is supposed to share
the parent process's address space until it exec's whereas fork does not.
But FreeBSD's updated VM model, which you can thank John Dyson for mostly,
is extremely efficient even when forking whole address spaces.

The differences between fork and vfork are also irrelevant in this
particular discussion since the fork/exec combination is only in
the 'fork' state for a very short time.  The moment you exec, the
address space is wiped ( except for mmap's MAP_INHERIT, which very few
people ever use ).

-Matt




To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Don Lewis
On Apr 15, 12:14pm, Peter Jeremy wrote:
} Subject: Re: swap-related problems
} Mikhail Teterin m...@misha.cisco.com wrote:
}  Worse then that,
} it may be possible to use it at malloc time, but unless your program
} runs and touches every page, the memory may not be available later.
} 
} If you run and touch every page, you are guaranteed to have the
} memory available, but you also increase the chances of you being the
} largest process when the system runs out of swap - in which case you
} get killed.

This could also has a pretty severe runtime performance penalty.  An
implementation that just reserves space would not.

} If we are up to discussing the possible implementations, I'd suggest
} that the system uses something other then SIGKILL to notify the
} program it's time to pay for the over-commit speed and convenience.
} I think, SIGBUS is appropriate, but I'm not sure.
} 
} I'm not sure this will gain a great deal.  Currently, if the kernel
} runs out of swap, it kills the largest runnable process.  For your
} proposal to work, it would need to kill the process that is requesting
} the space.  This raises a number of issues:
} 1) The problem is detected in vm_pageout_scan().  There's no obvious
}(to me anyway) way to locate the process that is triggering the
}space request.
} 2) The current approach kills the process hogging the greatest amount
}of memory.  This minimises the likelihood that you'll run out of
}swap again, quickly.
} 3) The process that triggered the request could potentially be in a
}non-runnable state.  In this case, the signal would be lost (or
}indefinitely delayed).
} 4) Since you're proposing a trap-able signal, the process may chose
}to ignore it and attempt to continue on.
} 5) The process would require additional stack space to process the
}signal (the signal handler frame and space for system calls to
}free memory as a minimum).
} 
} The last three issues could result in system deadlock.

Yes.  On the other hand, with an implemntation where malloc() returns
NULL, a carefully application could log a message and wait for more swap
to become available, or checkpoint itself.

} Having said all that, I agree that it would be useful if FreeBSD had a
} knob to disable overcommit - either on a system-wide, or per-process
} basis.  I don't feel sufficiently strongly about it to actually do
} something about it.  (From a quick look at the current code in
} vm_pageout_scan(), it would be fairly easy to add a per-process flag
} to prevent the process being a candidate for killing.  ptrace(2) or
} setrlimit(2) seem the most obvious ways to control the flag.  This
} would seem to allieviate the most common problem - one or two large,
} critical processes (eg the Xserver) getting killed, but probably has
} some nasty downside that I've overlooked).

Like if the Xserver has a memory leak, it will keep growing until most
of the other processes on the machine are killed of, maybe even important
ones, like the process that manipulates the control rods in the nuclear
reactor ;-)

Actually, that brings up a good point.  It is generally considered bad
practice for safety critical programs to use dynamic memory allocation
since it is so hard to guarantee that there won't be a memory allocation
failure.  With memory overcommit, it is possible that a process that
doesn't dynamically allocate memory to be killed because a fault could
happen when accessing BSS.



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-15 Thread Don Lewis
On Apr 14,  4:40pm, Chuck Robey wrote:
} Subject: Re: swap-related problems
} On Wed, 14 Apr 1999, Anthony Kimball wrote:

}  Well, it's only needed if you want to be able to reliably execute ANSI 
}  C code according to spec.  I personally don't care.  I'd be surprised
}  if core didn't though.  I would suspect that it would be deemed worthy
}  of someone's p2 queue, at least.
} 
} I can't understand this.  Make software that causes a major performance
} loss, and loses *bigtime* in memory allocation, just so the one guy to
} complain *at all* can not lose sleep over something that has causes no
} problems at all with any ANSI code in a properly sized system.

SunOS 4 doesn't do memory overcommit.  I've been running large processes
for years on SunOS 4 machines and haven't noticed any performance problems
due to lack of memory overcommit.  You just need to configure plenty of
swap.  I've found the old 3x RAM rule works fine, and it's really cheap
these days.  This could be shaved down a bit if SunOS didn't require
(swap  total VM) instead of (swap + RAM  total VM).

If you want to talk about slow, there are those crufty old implementations
that don't have COW fork().  If a large process forks, its entire memory
space needs to be duplicated, which is *really slow* if the process was
too big to fit in RAM to begin with.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



RE: swap-related problems

1999-04-14 Thread Ladavac Marino
 -Original Message-
 From: Mikhail Teterin [SMTP:m...@misha.cisco.com]
 Sent: Wednesday, April 14, 1999 12:45 AM
 To:   curr...@freebsd.org
 Subject:  Re: swap-related problems
 
 
 Well, this is just an implementation detail, is not it? I don't
 mean to critisize, or anything, but such thing as no available
 memory is a fairly intuitive... Coming down, again, the malloc
 should return a usable memory if available and NULL if it's not.
 Is not this a natural semantics? Why can a program die because
 _another_ program ate up all the rest of the memory?
 
 
[ML]  This is a common problem for any OS that implements memory
overcommit.  This means that it is not possible to detect an out-of-swap
condition sinchronously as the swap is reserved only when the pages are
dirtied and not when brk is grown.  This means that you can set brk a
gigabyte higher (given that your user limits allow that), and you will
not be using swap as long as you do not write to the pages you
allocated to the process.

Another strategy is to reserve the swap space as soon as it is
allocated by the program.  This strategy is much more conservative and
inherently safer, but it needs much more space: for instance, if you
have a program with WSS of a gigabyte and you want to system( date ),
you will need at least 2 gigs of swap because system() does fork() first
which means that you get 2 copies of your big program and the system
cannot know that in one of the copies an exec() will be shortly
forthcoming--thus, it has to reserve the full WSS for the copy because
it will potentially write to all pages of its WSS.

It would be nice if memory overcommit were configurable (on-off,
or per process).

/Marino


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



RE: swap-related problems

1999-04-14 Thread Alfred Perlstein
On Wed, 14 Apr 1999, Ladavac Marino wrote:

  -Original Message-
  From:   Mikhail Teterin [SMTP:m...@misha.cisco.com]
  Sent:   Wednesday, April 14, 1999 12:45 AM
  To: curr...@freebsd.org
  Subject:Re: swap-related problems
  
  
  Well, this is just an implementation detail, is not it? I don't
  mean to critisize, or anything, but such thing as no available
  memory is a fairly intuitive... Coming down, again, the malloc
  should return a usable memory if available and NULL if it's not.
  Is not this a natural semantics? Why can a program die because
  _another_ program ate up all the rest of the memory?
  
  
   [ML]  This is a common problem for any OS that implements memory
 overcommit.  This means that it is not possible to detect an out-of-swap
 condition sinchronously as the swap is reserved only when the pages are
 dirtied and not when brk is grown.  This means that you can set brk a
 gigabyte higher (given that your user limits allow that), and you will
 not be using swap as long as you do not write to the pages you
 allocated to the process.
 
   Another strategy is to reserve the swap space as soon as it is
 allocated by the program.  This strategy is much more conservative and
 inherently safer, but it needs much more space: for instance, if you
 have a program with WSS of a gigabyte and you want to system( date ),
 you will need at least 2 gigs of swap because system() does fork() first
 which means that you get 2 copies of your big program and the system
 cannot know that in one of the copies an exec() will be shortly
 forthcoming--thus, it has to reserve the full WSS for the copy because
 it will potentially write to all pages of its WSS.

An interesting idea would be to mark this process as killable if COW
causes an out of swap condition.  Another interesting application would
be a fork() call that checks this condition and fails if there is
potential for overcommit.  forkifavail()

Basically anyone doing a system(date); should be using vfork (yes
i can see when vfork is not sufficient)

This would sort of be like a soft limit on memory allocation and hint
to the kernel of which processes are the ones causeing overcommit.

basically at a certain point, sbrk will fail and forked processes
would be marked killable...

does this make sense?

has the idea of processes running with uid  10 or just root being
excempt from this frantic kill ?

-Alfred

 
   It would be nice if memory overcommit were configurable (on-off,
 or per process).
 
   /Marino
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

Alfred Perlstein - Admin, coder, and admirer of all things BSD.
-- There are operating systems, and then there's FreeBSD.
-- http://www.freebsd.org/4.0-current



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Poul-Henning Kamp
In message 199904132245.saa93...@misha.cisco.com, Mikhail Teterin writes:
Poul-Henning Kamp once wrote:

Well, this is just an implementation detail, is not it? I don't
mean to critisize, or anything, but such thing as no available
memory is a fairly intuitive... Coming down, again, the malloc
should return a usable memory if available and NULL if it's not.
Is not this a natural semantics? Why can a program die because
_another_ program ate up all the rest of the memory?

You know, this strikes me about as productive a discussion as the
split infinitive should be outlawed by style(9) we have every
so often.

Very very fundamental to UNIX philosophy is the maxim that it is
roots responsibility to configure the system right.

FreeBSD will defend itself against misconfigurations as best it
can, this includes shooting processes down when things get too
squeezy.

Think of it as self-defence.

The real problem is that you system isn't configured for what you
use it for.

--
Poul-Henning Kamp FreeBSD coreteam member
p...@freebsd.org   Real hackers run -current on their laptop.
FreeBSD -- It will take a long time before progress goes too far!


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



RE: swap-related problems

1999-04-14 Thread Brian Feldman
On Wed, 14 Apr 1999, Alfred Perlstein wrote:

 On Wed, 14 Apr 1999, Ladavac Marino wrote:
 
   -Original Message-
   From: Mikhail Teterin [SMTP:m...@misha.cisco.com]
   Sent: Wednesday, April 14, 1999 12:45 AM
   To:   curr...@freebsd.org
   Subject:  Re: swap-related problems
   
   
   Well, this is just an implementation detail, is not it? I don't
   mean to critisize, or anything, but such thing as no available
   memory is a fairly intuitive... Coming down, again, the malloc
   should return a usable memory if available and NULL if it's not.
   Is not this a natural semantics? Why can a program die because
   _another_ program ate up all the rest of the memory?
   
   
  [ML]  This is a common problem for any OS that implements memory
  overcommit.  This means that it is not possible to detect an out-of-swap
  condition sinchronously as the swap is reserved only when the pages are
  dirtied and not when brk is grown.  This means that you can set brk a
  gigabyte higher (given that your user limits allow that), and you will
  not be using swap as long as you do not write to the pages you
  allocated to the process.
  
  Another strategy is to reserve the swap space as soon as it is
  allocated by the program.  This strategy is much more conservative and
  inherently safer, but it needs much more space: for instance, if you
  have a program with WSS of a gigabyte and you want to system( date ),
  you will need at least 2 gigs of swap because system() does fork() first
  which means that you get 2 copies of your big program and the system
  cannot know that in one of the copies an exec() will be shortly
  forthcoming--thus, it has to reserve the full WSS for the copy because
  it will potentially write to all pages of its WSS.
 
 An interesting idea would be to mark this process as killable if COW
 causes an out of swap condition.  Another interesting application would
 be a fork() call that checks this condition and fails if there is
 potential for overcommit.  forkifavail()
 
 Basically anyone doing a system(date); should be using vfork (yes
 i can see when vfork is not sufficient)
 
 This would sort of be like a soft limit on memory allocation and hint
 to the kernel of which processes are the ones causeing overcommit.
 
 basically at a certain point, sbrk will fail and forked processes
 would be marked killable...
 
 does this make sense?
 
 has the idea of processes running with uid  10 or just root being
 excempt from this frantic kill ?

This isn't really a great idea since it would be changing the Unix
API. However, I agree that processes with exclusive use of the video
hardware  should not be killed by the system *COUGH XFree86 COUGH*.

 
 -Alfred
 
  
  It would be nice if memory overcommit were configurable (on-off,
  or per process).
  
  /Marino
  
  
  To Unsubscribe: send mail to majord...@freebsd.org
  with unsubscribe freebsd-current in the body of the message
  
 
 Alfred Perlstein - Admin, coder, and admirer of all things BSD.
 -- There are operating systems, and then there's FreeBSD.
 -- http://www.freebsd.org/4.0-current
 
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

 Brian Feldman_ __ ___   ___ ___ ___  
 gr...@unixhelp.org_ __ ___ | _ ) __|   \ 
 FreeBSD: The Power to Serve!  _ __ | _ \__ \ |) |
 http://www.freebsd.org   _ |___/___/___/ 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Brian Feldman
On Wed, 14 Apr 1999, Poul-Henning Kamp wrote:

 In message 199904132245.saa93...@misha.cisco.com, Mikhail Teterin writes:
 Poul-Henning Kamp once wrote:
 
 Well, this is just an implementation detail, is not it? I don't
 mean to critisize, or anything, but such thing as no available
 memory is a fairly intuitive... Coming down, again, the malloc
 should return a usable memory if available and NULL if it's not.
 Is not this a natural semantics? Why can a program die because
 _another_ program ate up all the rest of the memory?
 
 You know, this strikes me about as productive a discussion as the
 split infinitive should be outlawed by style(9) we have every
 so often.
 
 Very very fundamental to UNIX philosophy is the maxim that it is
 roots responsibility to configure the system right.
 
 FreeBSD will defend itself against misconfigurations as best it
 can, this includes shooting processes down when things get too
 squeezy.
 
 Think of it as self-defence.
 
 The real problem is that you system isn't configured for what you
 use it for.

Fine. Let's have a kernel option for it, because a LOT of us want the
old functionality back, and yes it _WAS_ there before.

 
 --
 Poul-Henning Kamp FreeBSD coreteam member
 p...@freebsd.org   Real hackers run -current on their laptop.
 FreeBSD -- It will take a long time before progress goes too far!
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

 Brian Feldman_ __ ___   ___ ___ ___  
 gr...@unixhelp.org_ __ ___ | _ ) __|   \ 
 FreeBSD: The Power to Serve!  _ __ | _ \__ \ |) |
 http://www.freebsd.org   _ |___/___/___/ 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Mikhail Teterin
Poul-Henning Kamp once stated:

=Well, this is just an implementation detail, is not it? I don't
=mean to critisize, or anything, but such thing as no available
=memory is a fairly intuitive... Coming down, again, the malloc
=should return a usable memory if available and NULL if it's not.
=Is not this a natural semantics? Why can a program die because
=_another_ program ate up all the rest of the memory?
=
=You know, this strikes me about as productive a discussion as the
[...]
=Very very fundamental to UNIX philosophy is the maxim that it is
=roots responsibility to configure the system right.

I'm sorry I managed to annoy you. However, a program needs to be
able to know if it can legally ask for more memory, right? And it
is very fundamental to malloc philosophy, that malloc returns
NULL, when it can not get more memory. Which it apparently does
now on FreeBSD, but only if the program exceeds an artificial
datasize limit...

And if it is not set, it the administrator's problem? May be...

Should we, perhaps, have it documented in a number of places,
including malloc(3)?

My other problem with the artificial limits, is that I can only
specify the absolute figures. Besides an ability to use Mb, Gb,
Kb, I'd like to have % there... And, as I mentioned before, an
ability to distribute the class database over NIS...

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Daniel C. Sobral
Ladavac Marino wrote:
 
 Another strategy is to reserve the swap space as soon as it is
 allocated by the program.  This strategy is much more conservative and
 inherently safer, but it needs much more space: for instance, if you
 have a program with WSS of a gigabyte and you want to system( date ),
 you will need at least 2 gigs of swap because system() does fork() first
 which means that you get 2 copies of your big program and the system
 cannot know that in one of the copies an exec() will be shortly
 forthcoming--thus, it has to reserve the full WSS for the copy because
 it will potentially write to all pages of its WSS.
 
 It would be nice if memory overcommit were configurable (on-off,
 or per process).

On AIX, you can have it set as a global option and on/off per
process. In my experience, though, setting it to off never solved
anything: if you need memory, you have to add memory.

--
Daniel C. Sobral(8-DCS)
d...@newsguy.com
d...@freebsd.org

nothing better than the ability to perform cunning linguistics




To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Chuck Robey
On Wed, 14 Apr 1999, Mikhail Teterin wrote:

 Poul-Henning Kamp once stated:
 
 =Well, this is just an implementation detail, is not it? I don't
 =mean to critisize, or anything, but such thing as no available
 =memory is a fairly intuitive... Coming down, again, the malloc
 =should return a usable memory if available and NULL if it's not.
 =Is not this a natural semantics? Why can a program die because
 =_another_ program ate up all the rest of the memory?
 =
 =You know, this strikes me about as productive a discussion as the
 [...]
 =Very very fundamental to UNIX philosophy is the maxim that it is
 =roots responsibility to configure the system right.
 
 I'm sorry I managed to annoy you. However, a program needs to be
 able to know if it can legally ask for more memory, right? And it
 is very fundamental to malloc philosophy, that malloc returns
 NULL, when it can not get more memory. Which it apparently does
 now on FreeBSD, but only if the program exceeds an artificial
 datasize limit...

He's not talking about an artificial limit, he's talking about another
user making off with all the memory.  This sounds very bizarre, how can
you possibly ask the system to predict what other user's are going to
ask for, in advance?  You can't possibly get absolute permissions on
malloc'ed area unless you reserve it (and thereby deny the area to other
users, when you might only use a small portion of your reserved area
yourself) or you restrict other's use of malloc'ed memory.

In your request, less users would have their requests satisfied, because
if they allocate 100 MB arrays (where they fill only 1KB with data) they
would actually be hard-reserving all that 100MB.  This doesn't sound
like any reasonable policy for general purpose use (although it might
very well suit your purposes).

These are fundamental design problems (which you have been told already)
and the way that FreeBSD chooses to act is arguably a very good one.
There is nothing here that violates any fundamental rules, because we
are talking about OS wide allocation strategies.  If you need
guarantees, then you have to buy more memory, more swap, and have less
hungry users.

 
 And if it is not set, it the administrator's problem? May be...
 
 Should we, perhaps, have it documented in a number of places,
 including malloc(3)?
 
 My other problem with the artificial limits, is that I can only
 specify the absolute figures. Besides an ability to use Mb, Gb,
 Kb, I'd like to have % there... And, as I mentioned before, an
 ability to distribute the class database over NIS...
 
   -mi
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

+---
Chuck Robey | Interests include any kind of voice or data 
chu...@picnic.mat.net   | communications topic, C programming, and Unix.
213 Lakeside Drive Apt T-1  |
Greenbelt, MD 20770 | I run picnic (FreeBSD-current)
(301) 220-2114  | and jaunt (Solaris7).
+---






To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Mikhail Teterin
Chuck Robey once stated:

=He's not talking about an artificial limit, he's talking about another
=user making off with all the memory.  This sounds very bizarre, how can
=you possibly ask the system to predict what other user's are going to
=ask for, in advance?  You can't possibly get absolute permissions on
[...]
=These are fundamental design problems (which you have been told already)
=and the way that FreeBSD chooses to act is arguably a very good one.
=There is nothing here that violates any fundamental rules, because we
=are talking about OS wide allocation strategies.  If you need
=guarantees, then you have to buy more memory, more swap, and have less
=hungry users.

All I want is that a program gets NULL from malloc if there is no memory
available. I find that to be a very fundamental thing about malloc.

In response, me and others are told at different times:

. there is no such thing as no memory available (!!!)
. you can get that behaviour by limiting the user's maximum
  datasize
. this wouldn't work without dumping the overcommit strategy,
  which would demand more memory and will slow things down
. be a better sysadmin
. get more memory and swap
. this discussion is annoying and fruitless (!!!)

I find this responses unacceptable, or only partially acceptable...

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Daniel C. Sobral
Mikhail Teterin wrote:
 
 All I want is that a program gets NULL from malloc if there is no memory
 available. I find that to be a very fundamental thing about malloc.
 
 In response, me and others are told at different times:
 
 . there is no such thing as no memory available (!!!)
 . you can get that behaviour by limiting the user's maximum
   datasize
 . this wouldn't work without dumping the overcommit strategy,
   which would demand more memory and will slow things down
 . be a better sysadmin
 . get more memory and swap
 . this discussion is annoying and fruitless (!!!)
 
 I find this responses unacceptable, or only partially acceptable...

There is two ways you can alloc memory: on-demand or pre-allocated.
Most Unixes out there choose on-demand because it let you do more
with less. A *lot* more. The only way to do what you want is to
pre-allocate.

Alas, suppose you set pre-allocation for your eat-all-memory
application. You application starts, eat all memory and stabilizes.
Then you move the mouse in X Free causing previously allocated
memory to be accessed. There is no memory available, so the largest
process is killed. That would be your eat-all-memory application.

What some of us have been telling you is that there is no easy way
out of this. Some have suggested don't kill this process
solutions. This is a poor solution, because it requires much more
sysadmin tweaking than limiting user's maximum datasize. 

If you pre-allocate all memory you'll find out you have much less
memory available for you. MUCH less.

Yeah, a pre-allocate all memory general system flag would be
interesting, but mostly because we could then say turn this on and
see how you suddenly don't have memory to even start X Free. Or, in
other words, if you set a maximum datasize for your application, you
can allocate MUCH more memory to that application than if our
allocation strategy was pre-allocate.

Do you have a solution? We don't.

--
Daniel C. Sobral(8-DCS)
d...@newsguy.com
d...@freebsd.org

nothing better than the ability to perform cunning linguistics


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Poul-Henning Kamp
In message 199904141314.jaa24...@kot.ne.mediaone.net, Mikhail Teterin writes:
Poul-Henning Kamp once stated:

=Well, this is just an implementation detail, is not it? I don't
=mean to critisize, or anything, but such thing as no available
=memory is a fairly intuitive... Coming down, again, the malloc
=should return a usable memory if available and NULL if it's not.
=Is not this a natural semantics? Why can a program die because
=_another_ program ate up all the rest of the memory?
=
=You know, this strikes me about as productive a discussion as the
[...]
=Very very fundamental to UNIX philosophy is the maxim that it is
=roots responsibility to configure the system right.

I'm sorry I managed to annoy you. However, a program needs to be
able to know if it can legally ask for more memory, right? And it
is very fundamental to malloc philosophy, that malloc returns
NULL, when it can not get more memory. Which it apparently does
now on FreeBSD, but only if the program exceeds an artificial
datasize limit...

malloc() on FreeBSD returns NULL when it cannot allocate the memory
asked for.

If you have an example where this is not the case I would VERY
much like to see it.

--
Poul-Henning Kamp FreeBSD coreteam member
p...@freebsd.org   Real hackers run -current on their laptop.
FreeBSD -- It will take a long time before progress goes too far!


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Mikhail Teterin
Poul-Henning Kamp once stated:

=malloc() on FreeBSD returns NULL when it cannot allocate the memory
=asked for.

=If you have an example where this is not the case I would VERY
=much like to see it.

I believe, a number of examples were given, when the use of the non-NULL
pointer returned by malloc resulted in the program being killed.

Those who asked why, were given an answer:

. because you allocated more memory then you have

They then, rightfully exclaimed:

. but should not malloc() have returned me NULL?

And the rest is drowned in the explanations of why it is very hard to assure.
I'm repeating myself here and so do others. We are probably facing the major
disadvantange of the overcommit strategy, and there may not, indeed, be a
way around this.

Documenting this properly is what is definitely in order...

While we are at this, what is the right way to find out if the returned pointer
can be used?

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Poul-Henning Kamp
In message 199904141501.laa25...@kot.ne.mediaone.net, Mikhail Teterin writes:
Poul-Henning Kamp once stated:

=malloc() on FreeBSD returns NULL when it cannot allocate the memory
=asked for.

=If you have an example where this is not the case I would VERY
=much like to see it.

I believe, a number of examples were given, when the use of the non-NULL
pointer returned by malloc resulted in the program being killed.

No, the examples I have seen showed the system killing a (somewhat)
random large process in an attempt to avoid deadlock.

I guess you can interpret it the way above if you insist on using a
very narrow application oriented focus.

Ignoring the larger issues is not a modus operandi I subscribe to.

Anyway, what you're asking for is really resource reservation.

Sorry FreeBSD doesn't support resource reservation for memory.

--
Poul-Henning Kamp FreeBSD coreteam member
p...@freebsd.org   Real hackers run -current on their laptop.
FreeBSD -- It will take a long time before progress goes too far!


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



RE: swap-related problems

1999-04-14 Thread Ladavac Marino
 -Original Message-
 From: Daniel C. Sobral [SMTP:d...@newsguy.com]
 Sent: Wednesday, April 14, 1999 3:04 PM
 To:   Ladavac Marino
 Cc:   'm...@aldan.algebra.com'; curr...@freebsd.org
 Subject:  Re: swap-related problems
 
 Ladavac Marino wrote:
  
  Another strategy is to reserve the swap space as soon as it
 is
  allocated by the program.  This strategy is much more conservative
 and
  inherently safer, but it needs much more space: for instance, if you
  have a program with WSS of a gigabyte and you want to system( date
 ),
  you will need at least 2 gigs of swap because system() does fork()
 first
  which means that you get 2 copies of your big program and the system
  cannot know that in one of the copies an exec() will be shortly
  forthcoming--thus, it has to reserve the full WSS for the copy
 because
  it will potentially write to all pages of its WSS.
  
  It would be nice if memory overcommit were configurable
 (on-off,
  or per process).
 
 On AIX, you can have it set as a global option and on/off per
 process. In my experience, though, setting it to off never solved
 anything: if you need memory, you have to add memory.
 
[ML]  Oh, memory overcommit does have its applications.
Remember the olden days of FORTRAN when dynamic memory allocation was
a meaningless term :)  Overcommit let the people allocate a
1*1 matrix and use only a 20*20 subset of it and have program
execute instead of fail out-of-swap.

Nowadays, vfork() could solve most of the problems on fork/exec.
Sadly, a frightening number of unices implement vfork() as

#define vfork fork

/Marino
 --
 Daniel C. Sobral  (8-DCS)
 d...@newsguy.com
 d...@freebsd.org
 
   nothing better than the ability to perform cunning linguistics
 


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



RE: swap-related problems

1999-04-14 Thread Ladavac Marino
 -Original Message-
 From: Mikhail Teterin [SMTP:m...@kot.ne.mediaone.net]
 Sent: Wednesday, April 14, 1999 5:01 PM
 To:   curr...@freebsd.org
 Subject:  Re: swap-related problems
 
 They then, rightfully exclaimed:
 
   . but should not malloc() have returned me NULL?
 
 And the rest is drowned in the explanations of why it is very hard to
 assure.
 I'm repeating myself here and so do others. We are probably facing the
 major
 disadvantange of the overcommit strategy, and there may not, indeed,
 be a
 way around this.
 
[ML]  Short of disabling memory overcommit, no there is no
answer.  If you do disable overcommit, prepare to add humongous swap
areas (or maybe not--some unices reserve the pages in the filesystem,
stealing the free blocks, so to say, but write the dirty pages to the
swap partitions.  They also keep 2 counters: swap pages reserved, and
swap pages actually used.  They are capable of swapping into a
filesystem should the need actually arise, but are very inefficient in
doing so.  However, having both numbers--swap reserved and swap
used--allows the sysadmin to properly size the swap partitions).

 Documenting this properly is what is definitely in order...
 
[ML]  I would agree to this.  Perhaps in the IMPLEMENTATION
section of the corresponding manpages (malloc, s/brk, fork.)

 While we are at this, what is the right way to find out if the
 returned pointer
 can be used?
 
[ML]  Sadly, in the memory overcommit situation, there is no way
to know whether a pointer returned by malloc will cause a process demise
or not.  The pointer is valid, but the swap area mapping is defered
until the page is dirtied (basically, the pointer points to a readonly
zero filled physical page and on write the trap handler tries to
allocate a backing swap area for the page.  If the swap is exhausted,
the handler eventually fails.  What the system does at this time is
irrelevant.

Please note that memory overcommit architectures are a rather
common optimization; FreeBSD is one of them.  They do, however, break
the ISO/ANSI C conformance (strictly speaking).

/Marino
   -mi
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Mikhail Teterin
Ladavac Marino once wrote:

  [ML] Sadly, in the memory overcommit situation, there is no way to
 know whether a pointer returned by malloc will cause a process demise
 or not. The pointer is valid, but the swap area mapping is defered
 until the page is dirtied (basically, the pointer points to a readonly
 zero filled physical page and on write the trap handler tries to
 allocate a backing swap area for the page. If the swap is exhausted,
 the handler eventually fails. What the system does at this time is
 irrelevant.
[...]
  Please note that memory overcommit architectures are a rather common
 optimization; FreeBSD is one of them. They do, however, break the
 ISO/ANSI C conformance (strictly speaking).

Aha, now its clearer. May be, since we are do not conform anyway,
we can design some clever way of notifying a program rather then
SIGKILL-ing it?

Perhaps, SIGBUS? Something, a program can catch, if it is prepared
to, and, may be, do some syscall to find out which chunk of memory
can not actually be used by it...

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Anthony Kimball

:  All I want is that a program gets NULL from malloc if there is no memory
:  available. I find that to be a very fundamental thing about malloc.

: Do you have a solution? We don't.

Make an sbrk variant which will pre-allocate backing store.
setenv MALLOC_PREALLOCATE

Not so hard.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Poul-Henning Kamp
In message 14100.61923.427423.153...@avalon.east, Anthony Kimball writes:

:  All I want is that a program gets NULL from malloc if there is no memory
:  available. I find that to be a very fundamental thing about malloc.

: Do you have a solution? We don't.

Make an sbrk variant which will pre-allocate backing store.
setenv MALLOC_PREALLOCATE

Not so hard.

1. Demonstrate the need.

2. Implement it

3. Send patches.

--
Poul-Henning Kamp FreeBSD coreteam member
p...@freebsd.org   Real hackers run -current on their laptop.
FreeBSD -- It will take a long time before progress goes too far!


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Anthony Kimball
Quoth Poul-Henning Kamp on Wed, 14 April:
:
: 1. Demonstrate the need.

Well, it's only needed if you want to be able to reliably execute ANSI 
C code according to spec.  I personally don't care.  I'd be surprised
if core didn't though.  I would suspect that it would be deemed worthy
of someone's p2 queue, at least.

: 2. Implement it
: 
: 3. Send patches.

And I certainly don't care enough to do that!-)


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Chuck Robey
On Wed, 14 Apr 1999, Anthony Kimball wrote:

 Quoth Poul-Henning Kamp on Wed, 14 April:
 :
 : 1. Demonstrate the need.
 
 Well, it's only needed if you want to be able to reliably execute ANSI 
 C code according to spec.  I personally don't care.  I'd be surprised
 if core didn't though.  I would suspect that it would be deemed worthy
 of someone's p2 queue, at least.

I can't understand this.  Make software that causes a major performance
loss, and loses *bigtime* in memory allocation, just so the one guy to
complain *at all* can not lose sleep over something that has causes no
problems at all with any ANSI code in a properly sized system.

The complainant could not notice a problem in any program that didn't
seriously over-allocate memory in the first place.  His program is far
more likely to work in our present paradigm than in the one he wishes
everyone to use.  I don't think he understands, at all, the global
effects of his requests; he has blinders on, and is looking at one
program, only, in a otherwise unloaded system.

This is supposed to be important enough to hit someone's work queue?  No
wonder *you* won't do it.


 
 : 2. Implement it
 : 
 : 3. Send patches.
 
 And I certainly don't care enough to do that!-)
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

+---
Chuck Robey | Interests include any kind of voice or data 
chu...@picnic.mat.net   | communications topic, C programming, and Unix.
213 Lakeside Drive Apt T-1  |
Greenbelt, MD 20770 | I run picnic (FreeBSD-current)
(301) 220-2114  | and jaunt (Solaris7).
+---






To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Anthony Kimball
Quoth Chuck Robey on Wed, 14 April:
: ... just so the one guy to
: complain *at all* 

Someone is always the first person to recognize a defect.  Until
others do, they stand alone.

: ...can not lose sleep over something that has causes no
: problems at all with any ANSI code in a properly sized system.

properly sized system?  Smirk.  Let's redirect this to chat.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Sheldon Hearn


On Wed, 14 Apr 1999 15:22:22 EST, Anthony Kimball wrote:

 : 3. Send patches.
 
 And I certainly don't care enough to do that!-)

So, what? You're just arguing for fun? If so, then you and everyone else
doing like wise can just piss the hell off.

Ciao,
Sheldon.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Anthony Kimball
Quoth Sheldon Hearn on Thu, 15 April:
: 
:  : 3. Send patches.
:  
:  And I certainly don't care enough to do that!-)
: 
: So, what? You're just arguing for fun? If so, then you and everyone else
: doing like wise can just piss the hell off.

[redirected to chat]

No, I'm trying to contribute constructively to a discussion which may
affect policy in the future.  One party suggested that there was an
admitted problem in the current system, but that we didn't know how
to fix it.  I suggested a fix.  This makes it easier for anyone who
cares enough to (1) write the code, and (2) fight the mugwump
politics, to implement a solution to the benefit of everyone who uses
the code.

Do I care about this particular problem that much?  No.  My concerns
lie elsewhere.  I do think it is important the the actual problem be
recognized, and that its solvability within acceptable constraints be
recognized.  Otherwise, people may make bad decisions -- for example,
rejecting patches offered by persons for whom this issue is more
pressing.

Besides, when someone makes a reasonable complaint and is derided
unfairly for it, it would be craven and despicable not to co-advocate.



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Andrew Reilly
On Wed, Apr 14, 1999 at 02:55:27PM -0500, Anthony Kimball wrote:
 
 :  All I want is that a program gets NULL from malloc if there is no memory
 :  available. I find that to be a very fundamental thing about malloc.
 
 : Do you have a solution? We don't.
 
 Make an sbrk variant which will pre-allocate backing store.
 setenv MALLOC_PREALLOCATE
 
 Not so hard.

You could do this yourself, I think, by:

1) create your own backing file of the appropriate size
2) mmap it into your address space
3) tell your own malloc() routine that that's where it should
   get memory from.

Any problems with that? (besides not being the system standard,
and the reduction of swapping efficiency caused by going through
the file system).

-- 
Andrew


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Mikhail Teterin
Sheldon Hearn once wrote:

  : 3. Send patches.
  
  And I certainly don't care enough to do that!-)

 So, what? You're just arguing for fun? If so, then you and everyone
 else doing like wise can just piss the hell off.

I, for one, was arguing because I saw a number of people ask about
this over the last years.

Instead of being told sorry, we do not comply with this part of
ANSI standard for reasons such and such, this people were told
how they have to be better sysadmins and what not.

This time around, it took several days of a long thread for somebody
to admit _the problem exists_.

People are sensitive about this -- you yourself had to resort to
using a notion refering to physiology AND a notion of religious
nature to express your feelings...

To avoid this discussion in the future, Handbook, FAQ, and malloc(3)
should be updated to reflect the fact, that FreeBSD's malloc does
NOT indeed to conform to ANSI/C, strictly speaking. All good
arguments as to why this is good for someone's health can also go
there. A TODO list can have an entry about this, because ANSI
compliance is A Good Thing(TM), is not it?

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Steve Kargl
Mikhail Teterin wrote:
 Sheldon Hearn once wrote:
 
   : 3. Send patches.
   
   And I certainly don't care enough to do that!-)
 
  So, what? You're just arguing for fun? If so, then you and everyone
  else doing like wise can just piss the hell off.
 
 I, for one, was arguing because I saw a number of people ask about
 this over the last years.
 
 Instead of being told sorry, we do not comply with this part of
 ANSI standard for reasons such and such, this people were told
 how they have to be better sysadmins and what not.
 
 This time around, it took several days of a long thread for somebody
 to admit _the problem exists_.
 

Then, submit your patches to fix the problem preferably with a knob
to turn your patches on/off.  From the discussion, I hope you're prepared
to deal with the following scenario:

  User:I compiled malloc with -DSTRICT_ANSI.  Now my system appears
   sluggish and I can't run as many processes because of
   memory/swap starvation.  

  Mikhail: Well, malloc() conforms to the ANSI C standard.  You need to
   buy more memory and a real big disk drive for swap space.
   Or, you could run fewer processes.

  User:But, if I compile malloc without -DSTRICT_ANSI, then everything
   seems to work.

  Mikhail: Well, that is silly because now your malloc doesn't conform
   to the ANSI C standard.  You should not use a nonconforming
   OS for programming.

Submit your patches to the documentation pointing out the alleged
shortcomings malloc() and friends.

Meanwhile, visit /usr/ports.

/usr/ports/devel/libxalloc
/usr/ports/devel/libmalloc
/usr/ports/devel/libdlmalloc

-- 
Steve


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Alfred Perlstein
On Thu, 15 Apr 1999, Andrew Reilly wrote:

 On Wed, Apr 14, 1999 at 02:55:27PM -0500, Anthony Kimball wrote:
  
  :  All I want is that a program gets NULL from malloc if there is no memory
  :  available. I find that to be a very fundamental thing about malloc.
  
  : Do you have a solution? We don't.
  
  Make an sbrk variant which will pre-allocate backing store.
  setenv MALLOC_PREALLOCATE
  
  Not so hard.
 
 You could do this yourself, I think, by:
 
 1) create your own backing file of the appropriate size
 2) mmap it into your address space
 3) tell your own malloc() routine that that's where it should
get memory from.
 
 Any problems with that? (besides not being the system standard,
 and the reduction of swapping efficiency caused by going through
 the file system).

actually doesn't the BSD mmap spec state that if you map anything
anon without special flags and there is no backing store available
that the call will fail, and that there is a way to pass a flag to tell
the system to overcommit but that you are prepared to take a SIGSEGV 
if you do cause the system to run out of backing store on the mepped object?

looking at the manpages it seems that this is either undocumented or
was never completed.

-Alfred




To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Mikhail Teterin
Steve Kargl once wrote:

 Then, submit your patches to fix the problem preferably with a knob to
 turn your patches on/off. From the discussion, I hope you're prepared
 to deal with the following scenario:
 
I'll be prepared... I don't think it should be a compile time option,
though. May be kernel-compile-time -- malloc isn't really at fault here,
it seems... But rather a run-time.

   User:I compiled malloc with -DSTRICT_ANSI.  Now my system appears
sluggish and I can't run as many processes because of
memory/swap starvation.  
[...]
 
 Submit your patches to the documentation pointing out the alleged
 shortcomings malloc() and friends.

Alleged :-)
 
 Meanwhile, visit /usr/ports.

 /usr/ports/devel/libxalloc
 /usr/ports/devel/libmalloc
 /usr/ports/devel/libdlmalloc

I do not see how they can guarantee the usability of the returned
memory with the current kernel. There apparently is no way of
knowing at malloc time if the memory can be used. Worse then that,
it may be possible to use it at malloc time, but unless your program
runs and touches every page, the memory may not be available later.

If we are up to discussing the possible implementations, I'd suggest
that the system uses something other then SIGKILL to notify the
program it's time to pay for the over-commit speed and convenience.
I think, SIGBUS is appropriate, but I'm not sure.

Anything a program can _catch_ and react upon, if its author chooses
to.

Then, one can write a safe malloc, which will install the signal
handler, and touch every page in the the memory referenced by the
to-be-returned pointer. If the signal handler is invoked in the
progress, the to-be-returned memory must be returned back to the
system and NULL should be returned to the caller.

Whether or not to use this safe malloc can be determined by malloc
options and/or the size of the request.

However, my (in)ability to propose anything remotely sensible does
not change the facts established in this painful thread. That our
malloc does not conform to standards (for whatever reasons), and
that something should be done about it. That something must start
with documenting the flaw...

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Alfred Perlstein
On Wed, 14 Apr 1999, Mikhail Teterin wrote:

 Steve Kargl once wrote:
 
  /usr/ports/devel/libxalloc
  /usr/ports/devel/libmalloc
  /usr/ports/devel/libdlmalloc
 
 I do not see how they can guarantee the usability of the returned
 memory with the current kernel. There apparently is no way of
 knowing at malloc time if the memory can be used. Worse then that,
 it may be possible to use it at malloc time, but unless your program
 runs and touches every page, the memory may not be available later.
 
 If we are up to discussing the possible implementations, I'd suggest
 that the system uses something other then SIGKILL to notify the
 program it's time to pay for the over-commit speed and convenience.
 I think, SIGBUS is appropriate, but I'm not sure.
 
 Anything a program can _catch_ and react upon, if its author chooses
 to.

Design an impelementation says a SIGSEGV should be delivered afaik,
but that's only for anon mmap.  the problem with that is that you then
can tell if your allocator is buggy, or you have a memory problem...

 Then, one can write a safe malloc, which will install the signal
 handler, and touch every page in the the memory referenced by the
 to-be-returned pointer. If the signal handler is invoked in the
 progress, the to-be-returned memory must be returned back to the
 system and NULL should be returned to the caller.
 
 Whether or not to use this safe malloc can be determined by malloc
 options and/or the size of the request.
 
 However, my (in)ability to propose anything remotely sensible does
 not change the facts established in this painful thread. That our
 malloc does not conform to standards (for whatever reasons), and
 that something should be done about it. That something must start
 with documenting the flaw...

It's an impelementation, NOT a flaw.  there should be a sysctl
to force your standard impelementation, this current one is
great for servers, desktop boxes and shell boxes, however for
embedded systems where compenents shouldn't fail being killed
for using memory you rightously allocated or potentially causing
the death of an entirly different programming running is unnaceptable.

it also makes for a really big PITA to debug the reason why a totally 
unrelated process was killed.

-Alfred




To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Brian Dean

 I do not see how they can guarantee the usability of the returned
 memory with the current kernel. There apparently is no way of
 knowing at malloc time if the memory can be used. Worse then that,
 it may be possible to use it at malloc time, but unless your program
 runs and touches every page, the memory may not be available later.

Check the man page (and the source).  It looks as if export
MALLOC_OPTIONS=Z will cause malloc to touch every page and zero it
out.  This may be in part what you are wanting.  Also, you can compile
in malloc_options = Z to acheive the same affect without having to
always export the environment variable.

Of course, when you do this, your process will (might) still be killed
immediately or at some point in the future, which I think is what you
are trying to avoid :(

-Brian
--
Brian Dean  brd...@unx.sas.com


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Brian Feldman
On Wed, 14 Apr 1999, Poul-Henning Kamp wrote:

 In message 199904141501.laa25...@kot.ne.mediaone.net, Mikhail Teterin 
 writes:
 Poul-Henning Kamp once stated:
 
 =malloc() on FreeBSD returns NULL when it cannot allocate the memory
 =asked for.
 
 =If you have an example where this is not the case I would VERY
 =much like to see it.
 
 I believe, a number of examples were given, when the use of the non-NULL
 pointer returned by malloc resulted in the program being killed.
 
 No, the examples I have seen showed the system killing a (somewhat)
 random large process in an attempt to avoid deadlock.
 
 I guess you can interpret it the way above if you insist on using a
 very narrow application oriented focus.
 
 Ignoring the larger issues is not a modus operandi I subscribe to.
 
 Anyway, what you're asking for is really resource reservation.
 
 Sorry FreeBSD doesn't support resource reservation for memory.

mlock()? BTW, why can't anyone explain why the old behavior WAS THAT
the process would get NULL? Now it's this...

 
 --
 Poul-Henning Kamp FreeBSD coreteam member
 p...@freebsd.org   Real hackers run -current on their laptop.
 FreeBSD -- It will take a long time before progress goes too far!
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

 Brian Feldman_ __ ___   ___ ___ ___  
 gr...@unixhelp.org_ __ ___ | _ ) __|   \ 
 FreeBSD: The Power to Serve!  _ __ | _ __ \ |) |
 http://www.freebsd.org   _ |___/___/___/ 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Matthew D. Fuller
On Wed, Apr 14, 1999 at 01:12:00PM -0400, a little birdie told me
that Mikhail Teterin remarked
 
 Aha, now its clearer. May be, since we are do not conform anyway,
 we can design some clever way of notifying a program rather then
 SIGKILL-ing it?
 
 Perhaps, SIGBUS? Something, a program can catch, if it is prepared
 to, and, may be, do some syscall to find out which chunk of memory
 can not actually be used by it...

I think you're missing a point here.  (Or maybe I am, I don't know that
much about it myself).  The program that gets killed is NOT necessarily
the program that tried to access something causing a memory overcommit;
rather, the program that gets killed is the largest program around.
Often, they'll probably coincide, but by no means necessarily.

You'd also have to rely on somebody not writing a program that, inside
the SIGBUS handler, allocates and touches more memory, causing a SIGBUS,
causing more allocation, etc etc etc.  Lets try not opening any more
DoSen than there already are, 'K?



---

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
| Matthew Fuller  http://www.over-yonder.net/ |
* fulle...@futuresouth.com   fulle...@over-yonder.net *
| UNIX Systems Administrator  Specializing in FreeBSD |
*   FutureSouth Communications   ISPHelp ISP Consulting   *
|  The only reason I'm burning my candle at both ends,   |
*is because I haven't figured out how to light the*
| middle yet |
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Brian Feldman
On Wed, 14 Apr 1999, Poul-Henning Kamp wrote:

 In message 14100.61923.427423.153...@avalon.east, Anthony Kimball writes:
 
 :  All I want is that a program gets NULL from malloc if there is no memory
 :  available. I find that to be a very fundamental thing about malloc.
 
 : Do you have a solution? We don't.
 
 Make an sbrk variant which will pre-allocate backing store.
 setenv MALLOC_PREALLOCATE
 
 Not so hard.
 
 1. Demonstrate the need.

I'm not doing this, damnit ;)

 
 2. Implement it

Add another character to the malloc options there already. When something is
allocated, touch the first char to be '\0'. Implementation done.

 
 3. Send patches.

Part of 2.

 
 --
 Poul-Henning Kamp FreeBSD coreteam member
 p...@freebsd.org   Real hackers run -current on their laptop.
 FreeBSD -- It will take a long time before progress goes too far!
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

 Brian Feldman_ __ ___   ___ ___ ___  
 gr...@unixhelp.org_ __ ___ | _ ) __|   \ 
 FreeBSD: The Power to Serve!  _ __ | _ __ \ |) |
 http://www.freebsd.org   _ |___/___/___/ 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Jim Bloom
A signal handler is not guaranteed to work.  It must be written such that it
does not require a new page of memory.  Some possible problems here are the
stack growing, writing on a new page in the data segment, etc.  I'm not familiar
enough with the VM system, but if you couldn't create a new swap page for a
process, what would guarantee that the signal handler could get the pages it
needs.

Jim Bloom
bl...@acm.org

Mikhail Teterin wrote:
 
 If we are up to discussing the possible implementations, I'd suggest
 that the system uses something other then SIGKILL to notify the
 program it's time to pay for the over-commit speed and convenience.
 I think, SIGBUS is appropriate, but I'm not sure.
 
 Anything a program can _catch_ and react upon, if its author chooses
 to.



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Brian Feldman
On Wed, 14 Apr 1999, Anthony Kimball wrote:

 Quoth Poul-Henning Kamp on Wed, 14 April:
 :
 : 1. Demonstrate the need.
 
 Well, it's only needed if you want to be able to reliably execute ANSI 
 C code according to spec.  I personally don't care.  I'd be surprised
 if core didn't though.  I would suspect that it would be deemed worthy
 of someone's p2 queue, at least.

ACTUALLY it would still break ANSI because the malloc itself would crash
the program, instead of touching the memory manually.

 
 : 2. Implement it
 : 
 : 3. Send patches.
 
 And I certainly don't care enough to do that!-)
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 

 Brian Feldman_ __ ___   ___ ___ ___  
 gr...@unixhelp.org_ __ ___ | _ ) __|   \ 
 FreeBSD: The Power to Serve!  _ __ | _ __ \ |) |
 http://www.freebsd.org   _ |___/___/___/ 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Jordan K. Hubbard
 I'm not doing this, damnit ;)

Then can we end this stupid thread already?

- Jordan


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Peter Jeremy
Mikhail Teterin m...@misha.cisco.com wrote:
 Worse then that,
it may be possible to use it at malloc time, but unless your program
runs and touches every page, the memory may not be available later.

If you run and touch every page, you are guaranteed to have the
memory available, but you also increase the chances of you being the
largest process when the system runs out of swap - in which case you
get killed.

If we are up to discussing the possible implementations, I'd suggest
that the system uses something other then SIGKILL to notify the
program it's time to pay for the over-commit speed and convenience.
I think, SIGBUS is appropriate, but I'm not sure.

I'm not sure this will gain a great deal.  Currently, if the kernel
runs out of swap, it kills the largest runnable process.  For your
proposal to work, it would need to kill the process that is requesting
the space.  This raises a number of issues:
1) The problem is detected in vm_pageout_scan().  There's no obvious
   (to me anyway) way to locate the process that is triggering the
   space request.
2) The current approach kills the process hogging the greatest amount
   of memory.  This minimises the likelihood that you'll run out of
   swap again, quickly.
3) The process that triggered the request could potentially be in a
   non-runnable state.  In this case, the signal would be lost (or
   indefinitely delayed).
4) Since you're proposing a trap-able signal, the process may chose
   to ignore it and attempt to continue on.
5) The process would require additional stack space to process the
   signal (the signal handler frame and space for system calls to
   free memory as a minimum).

The last three issues could result in system deadlock.

 That our
malloc does not conform to standards (for whatever reasons),
The problem is the system, rather than malloc.  I'm not sure that
the ANSI C standard forbids the current system behaviour.  The
behaviour is quite common, so I'm sure it's not forbidden by POSIX
or various other Unix specs.

that something should be done about it.
This is a commonly used phrase around here.  Basically, you have
3 choices:
1) Fix the problem yourself (ie submit patches that fix the problem)
2) Convince someone else that the problem is critical enough for them
   to fix it for you.
3) Pay someone to fix it for you (this is an extension of the previous
   item).

 That something must start
with documenting the flaw...
Feel free to write a PR.  Including the changes you want to the
documentation (in text format if you don't want to write a patch
to the man page) increases the chances of the changes getting done.

Having said all that, I agree that it would be useful if FreeBSD had a
knob to disable overcommit - either on a system-wide, or per-process
basis.  I don't feel sufficiently strongly about it to actually do
something about it.  (From a quick look at the current code in
vm_pageout_scan(), it would be fairly easy to add a per-process flag
to prevent the process being a candidate for killing.  ptrace(2) or
setrlimit(2) seem the most obvious ways to control the flag.  This
would seem to allieviate the most common problem - one or two large,
critical processes (eg the Xserver) getting killed, but probably has
some nasty downside that I've overlooked).

Peter


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Anthony Kimball
Quoth Brian Feldman on Wed, 14 April:
: 
: ACTUALLY it would still break ANSI because the malloc itself would crash
: the program, instead of touching the memory manually.

The point here is that malloc can return null when it fails to allocate.
I might mention that Andrew Reilly's suggestion is the winner so far,
in my mind.  The solution of having libc:malloc honor a 'setenv 
MALLOC_MMAP_STORE
/path/here' would work with shrink-wrap software, and completely
bypass the concerns, real and imagined, of even the most zealous 
advocate of overcommit purity.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Warner Losh
In message 199904142340.taa96...@misha.cisco.com Mikhail Teterin writes:
: Then, one can write a safe malloc, which will install the signal
: handler, and touch every page in the the memory referenced by the
: to-be-returned pointer. If the signal handler is invoked in the
: progress, the to-be-returned memory must be returned back to the
: system and NULL should be returned to the caller.

This won't work all the time.  FreeBSD overcommits swap space and you
may get a SIGKILL even if you've touched all the pages.  FreeBSD kills
processes when swap space runs out.

: However, my (in)ability to propose anything remotely sensible does
: not change the facts established in this painful thread. That our
: malloc does not conform to standards (for whatever reasons), and
: that something should be done about it. That something must start
: with documenting the flaw...

The behavior is documented:
 The malloc() and calloc() functions return a pointer to the allocated
 memory if successful; otherwise a NULL pointer is returned.

What the system does when it has resource shortages is beyond the
scope of the ANSI-C standard, so I don't see why you say that
FreeBSD's malloc isn't standard conforming.

Warner


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Mikhail Teterin
Warner Losh once stated:

=: Then, one can write a safe malloc, which will install the signal
=: handler, and touch every page in the the memory referenced by the
=: to-be-returned pointer. If the signal handler is invoked in the
=: progress, the to-be-returned memory must be returned back to the
=: system and NULL should be returned to the caller.

=This won't work all the time. FreeBSD overcommits swap space and you
=may get a SIGKILL even if you've touched all the pages. FreeBSD kills
=processes when swap space runs out.

That's my point! I advocate the use of some _other_ signal. Something
catchable.

=: However, my (in)ability to propose anything remotely sensible does
=: not change the facts established in this painful thread. That our
=: malloc does not conform to standards (for whatever reasons), and that
=: something should be done about it. That something must start with
=: documenting the flaw...

=The behavior is documented:
= The malloc() and calloc() functions return a pointer to the allocated
= memory if successful; otherwise a NULL pointer is returned.
=
=What the system does when it has resource shortages is beyond the scope
=of the ANSI-C standard, so I don't see why you say that FreeBSD's
=malloc isn't standard conforming.

In case of resource shortage the malloc should be unsuccessful
and return NULL. Instead, right now you may still get a non-NULL
pointer, but get a SIGKILL when you try to use the rightfully
allocated memory.

-mi



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Daniel C. Sobral
Anthony Kimball wrote:
 
 :  All I want is that a program gets NULL from malloc if there is no memory
 :  available. I find that to be a very fundamental thing about malloc.
 
 : Do you have a solution? We don't.
 
 Make an sbrk variant which will pre-allocate backing store.
 setenv MALLOC_PREALLOCATE

As soon as this application grabs all the memory, the next thing to
request more memory (not a malloc!) will cause the application to be
killed (largest application criteria). Change the criteria, and
*another* application gets killed. Any application getting killed is
a problem, you'll find very fast.

--
Daniel C. Sobral(8-DCS)
d...@newsguy.com
d...@freebsd.org

nothing better than the ability to perform cunning linguistics


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-14 Thread Peter Jeremy
Mikhail Teterin m...@kot.ne.mediaone.net wrote:
That's my point! I advocate the use of some _other_ signal. Something
catchable.

As soon as you allow a catchable signal, you create a potential
deadlock situation.  See my previous mail.

In case of resource shortage the malloc should be unsuccessful
and return NULL.

This requires a change to the kernel to disable swap over-commit.
malloc is behaving correctly: it calls brk/sbrk to request additional
memory.  The kernel verifies that the process hasn't exceeded its
resource limits and the brk returns success.

 Instead, right now you may still get a non-NULL
pointer, but get a SIGKILL when you try to use the rightfully
allocated memory.

Actually the biggest process gets SIGKILL.  This probably isn't the
process that requested the memory.

Peter


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-13 Thread Poul-Henning Kamp
In message 199904121505.laa70...@misha.cisco.com, Mikhail Teterin writes:
Matthew Dillon once wrote:
 
 If you unset the datasize limit and the program does not exceed
 the maximum system-supported datasize limit, malloc() should not
 return NULL even if the system is out of swap.

Can you explain why? Our malloc(3) is a little fuzzy as to the
circumstances under which malloc will fail, but on Solaris, it is
more explicit:

   RETURN VALUES
   If there is no available memory, malloc(), realloc(),
   memalign(), valloc(), and calloc() return a null pointer.

I consider being out-of-swap as having no available memory. Wouldn't you?

The trouble is, there is no such thing as no available memory.

What we have instead is not enough memory available for this specific
request.

For phkmalloc I can say in particular that there is a pool for each
size of chunk (16,32,64...2048 bytes and for pages.  If you run out
of one of these there may still be stuff in the other pools.

--
Poul-Henning Kamp FreeBSD coreteam member
p...@freebsd.org   Real hackers run -current on their laptop.
FreeBSD -- It will take a long time before progress goes too far!


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-13 Thread Mikhail Teterin
Poul-Henning Kamp once wrote:

  If you unset the datasize limit and the program does not exceed
  the maximum system-supported datasize limit, malloc() should
  not return NULL even if the system is out of swap.

 Can you explain why? Our malloc(3) is a little fuzzy as to the
 circumstances under which malloc will fail, but on Solaris, it is
 more explicit:

  RETURN VALUES
  If there is no available memory, malloc(), realloc(),
  memalign(), valloc(), and calloc() return a null pointer.
 
 I consider being out-of-swap as having no available memory. Wouldn't you?
 
 The trouble is, there is no such thing as no available memory.

Well, this is just an implementation detail, is not it? I don't
mean to critisize, or anything, but such thing as no available
memory is a fairly intuitive... Coming down, again, the malloc
should return a usable memory if available and NULL if it's not.
Is not this a natural semantics? Why can a program die because
_another_ program ate up all the rest of the memory?

 What we have instead is not enough memory available for this specific
 request.

 For phkmalloc I can say in particular that there is a pool for each
 size of chunk (16,32,64...2048 bytes and for pages. If you run out of
 one of these there may still be stuff in the other pools.

Then the malloc may need to take its time rearranging the pools,
or whatever... It gets it right if there is a datasize limit, right?
But the current behaviour under low-memory conditions seems wrong
to me, and is certainly different from what the Solaris' malloc(3C)
states.

-mi

DISCLAIMER:
a) this message was not meant to offend;
b) I realize, the Solaris' malloc() may also be broken
   in many ways;
c) FreeBSD is rightfully the OS of choice for many (including
   myself);
d) I realize, that FreeBSD is wonderfull and I myself
   use it _all_ i386 machines under my control and on
   most i386 machines outside of my control.


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-12 Thread Mikhail Teterin
Matthew Dillon once wrote:
 
 If you unset the datasize limit and the program does not exceed
 the maximum system-supported datasize limit, malloc() should not
 return NULL even if the system is out of swap.

Can you explain why? Our malloc(3) is a little fuzzy as to the
circumstances under which malloc will fail, but on Solaris, it is
more explicit:

RETURN VALUES
If there is no available memory, malloc(), realloc(),
memalign(), valloc(), and calloc() return a null pointer.

I consider being out-of-swap as having no available memory. Wouldn't you?

Now, for better diagnostics and for a possibility of smarter applications,
they also have:

ERRORS
 If malloc(), calloc(), or realloc() returns  unsuccessfully,
 errno will be set to indicate the following:

 ENOMEM size bytes of  memory  exceeds  the  physical
limits  of  your  system, and cannot be allo-
cated.

 EAGAIN There is not enough memory available AT  THIS
POINT  IN  TIME  to  allocate  size  bytes of
memory; but the application could  try  again
later.

 The correct solution is to set a maximum datasize limit. That is
 what it is there for. I generally unlimit the datasize for root
 and set it to around 64MB for all other users.

Wait, I want to give all the memory available to a program, but I
want it to be NOTIFIED when no more memory is left. True, I can
set the limit to the actual amount of my swap, but that would be
hard to maintain and still does not solve a problem of running
together with some other memory consuming application.

The existing situation is sort of like killing a program for trying
to write to disk in excess of the user's quota, or for being ready
to RUN when the load is 1.

-mi


To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-12 Thread Julian Elischer
It seems to me that you might kill the forst program the has a lot of ram
in unse that has not specified a limit for itself.. programs that are wel
enough behaved to set a limit for themselves should be rewarded.

just a thought...
julian


On Sun, 11 Apr 1999, Matthew Dillon wrote:

 :I use the memory as soon as it's malloced. If it reserves a page, then
 :pagefaults it into existence, the VM system knows that that page is now
 :allocated. When I malloc the last available page for user use, the VM
 :system knows that it's the last page. I dirty it, and there are none
 :free. If I malloc(), I want to know that there is no more memory, not
 :have my process killed. This is POMA.
 :
 :Previously, the POLA, a NULL getting returned, WORKED CORRECTLY. It did this
 :for a long time. My little test program always showed this, and shows
 :that now something was broken. I'll attach it to the end.
 
 I ran your program.  malloc() appears to work properly -- returns NULL 
 when
 the datasize limit is reached.  In my case, I set the datasize limit 
 to 64MB and ran the program.
 
 ...
 mallocs failed at 64956
 
 Under 4.0-current.
 
 : then one or two processes getting killed.  Having N random processes 
 get
 : malloc() failures can lead to serious instability with processes.
 :
 :Only bad code doesn't check return values of malloc().
 
 You are volunteering to run through the thousands of programs  ports
 to make sure that every one checks the return value for malloc()?
 
 Declarations that don't solve problems are not relevant.  It's bad enough
 that we have to kill something to handle an out-of-vm situation, we
 shouldn't go off and destabilize the rest of the system while we are at 
 it.
 
   -Matt
 
 
 
 To Unsubscribe: send mail to majord...@freebsd.org
 with unsubscribe freebsd-current in the body of the message
 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-11 Thread Matthew Dillon
:It seems that something has broken the good ol' swap behavior. For instance,
:I have my user-limits set to unlimited and I run something which uses up
:all RAM. Mallocing never FAILS in the program, as brk() doesn't fail, as etc
:etc etc. But mallocing continues, all swap space gets used, and both the
:runaway process and the next biggest gets killed (XF86, of course).
:  Matt, perhaps you can shed light on
:   a. why mallocs still succeed after
:swap_pager: out of swap space
:swap_pager_getswapspace: failed
:  is displayed
:   b. why the process continues and gets killed TWICE, or two different
:   processes get killed
:?
:
: Brian Feldman_ __ ___   ___ ___ ___  

The thing with the processes getting killed twice or two different
processes being killed is due to the latency between sending the signal
to kill the process and the process actually going away.  In order to be
killed, the process must be woken up and allowed to run.

malloc() still succeeds because it isn't actually touching the new memory,
and so the system doesn't actually reserve the memory as of the time of
the system call.  That is done by the user program later on after malloc()
returns.  malloc()'s limitations are based on the process resources, not
available memory.

It would not be appropriate to make malloc() fail in this situation because
this would result in N random programs getting malloc() failures rather
then one or two processes getting killed.  Having N random processes get
malloc() failures can lead to serious instability with processes.

What might help in a situation like this would a way to flag certain
processes as being 'very important' ... that should only be killed as
a last resort, even if they have a relatively large RSS.

-Matt
Matthew Dillon 
dil...@backplane.com



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-11 Thread Brian Feldman
On Sun, 11 Apr 1999, Matthew Dillon wrote:

 :It seems that something has broken the good ol' swap behavior. For instance,
 :I have my user-limits set to unlimited and I run something which uses up
 :all RAM. Mallocing never FAILS in the program, as brk() doesn't fail, as etc
 :etc etc. But mallocing continues, all swap space gets used, and both the
 :runaway process and the next biggest gets killed (XF86, of course).
 :  Matt, perhaps you can shed light on
 : a. why mallocs still succeed after
 :swap_pager: out of swap space
 :swap_pager_getswapspace: failed
 :is displayed
 : b. why the process continues and gets killed TWICE, or two different
 : processes get killed
 :?
 :
 : Brian Feldman_ __ ___   ___ ___ ___  
 
 The thing with the processes getting killed twice or two different
 processes being killed is due to the latency between sending the signal
 to kill the process and the process actually going away.  In order to be
 killed, the process must be woken up and allowed to run.

Then let's make sure the kill waits before it does that.

 
 malloc() still succeeds because it isn't actually touching the new memory,
 and so the system doesn't actually reserve the memory as of the time of
 the system call.  That is done by the user program later on after malloc()
 returns.  malloc()'s limitations are based on the process resources, not
 available memory.

I use the memory as soon as it's malloced. If it reserves a page, then
pagefaults it into existence, the VM system knows that that page is now
allocated. When I malloc the last available page for user use, the VM
system knows that it's the last page. I dirty it, and there are none
free. If I malloc(), I want to know that there is no more memory, not
have my process killed. This is POMA.

Previously, the POLA, a NULL getting returned, WORKED CORRECTLY. It did this
for a long time. My little test program always showed this, and shows
that now something was broken. I'll attach it to the end.

 
 It would not be appropriate to make malloc() fail in this situation 
 because
 this would result in N random programs getting malloc() failures rather
 then one or two processes getting killed.  Having N random processes get
 malloc() failures can lead to serious instability with processes.

Only bad code doesn't check return values of malloc().

 
 What might help in a situation like this would a way to flag certain
 processes as being 'very important' ... that should only be killed as
 a last resort, even if they have a relatively large RSS.

Yes, I was thinking of something like this for the X server.

 
   -Matt
   Matthew Dillon 
   dil...@backplane.com
 
 

 Brian Feldman_ __ ___   ___ ___ ___  
 gr...@unixhelp.org_ __ ___ | _ ) __|   \ 
 FreeBSD: The Power to Serve!  _ __ | _ \__ \ |) |
 http://www.freebsd.org   _ |___/___/___/ 


ts=4

#include stdio.h
#include stdlib.h
#include string.h
#define SIZE 1024*1024

#ifdef OTHER_VER
const unsigned int junk[] = {
0xdeadbeef,
0xbeeff00d,
0x133751ff,
0xf33db34f
};
#endif

void
main(void) {
int count, yep = 0;
void *stfu[SIZE];
void *mem;

for (count = 0; count  SIZE; count++) {
if ((mem = stfu[count] = malloc(1024))) {
int where;
printf(%p (%i) malloc'd\n, stfu[count], count);
for (where = 0; where  (1024 / sizeof(unsigned)); 
where++)
((unsigned *)mem)[where] =
#ifdef OTHER_VER

junk[
#endif

where
#ifdef OTHER_VER

% 4]
#endif

;
yep++;
}   
else
break;
}
/*  free(stfu[yep--]); */
for (count = 0; count  yep; count++) {
int where;

mem = stfu[count];
for (where = 0; where  (1024 / sizeof(unsigned)); where++)
if (((unsigned *)mem)[where] !=
#ifdef OTHER_VER

junk[
#endif

where
#ifdef OTHER_VER
  

Re: swap-related problems

1999-04-11 Thread Matthew Dillon
:I use the memory as soon as it's malloced. If it reserves a page, then
:pagefaults it into existence, the VM system knows that that page is now
:allocated. When I malloc the last available page for user use, the VM
:system knows that it's the last page. I dirty it, and there are none
:free. If I malloc(), I want to know that there is no more memory, not
:have my process killed. This is POMA.
:
:Previously, the POLA, a NULL getting returned, WORKED CORRECTLY. It did this
:for a long time. My little test program always showed this, and shows
:that now something was broken. I'll attach it to the end.

I ran your program.  malloc() appears to work properly -- returns NULL when
the datasize limit is reached.  In my case, I set the datasize limit 
to 64MB and ran the program.

...
mallocs failed at 64956

Under 4.0-current.

: then one or two processes getting killed.  Having N random processes get
: malloc() failures can lead to serious instability with processes.
:
:Only bad code doesn't check return values of malloc().

You are volunteering to run through the thousands of programs  ports
to make sure that every one checks the return value for malloc()?

Declarations that don't solve problems are not relevant.  It's bad enough
that we have to kill something to handle an out-of-vm situation, we
shouldn't go off and destabilize the rest of the system while we are at it.

-Matt



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-11 Thread Brian Feldman
On Sun, 11 Apr 1999, Matthew Dillon wrote:

 :I use the memory as soon as it's malloced. If it reserves a page, then
 :pagefaults it into existence, the VM system knows that that page is now
 :allocated. When I malloc the last available page for user use, the VM
 :system knows that it's the last page. I dirty it, and there are none
 :free. If I malloc(), I want to know that there is no more memory, not
 :have my process killed. This is POMA.
 :
 :Previously, the POLA, a NULL getting returned, WORKED CORRECTLY. It did this
 :for a long time. My little test program always showed this, and shows
 :that now something was broken. I'll attach it to the end.
 
 I ran your program.  malloc() appears to work properly -- returns NULL 
 when
 the datasize limit is reached.  In my case, I set the datasize limit 
 to 64MB and ran the program.

Unset the datasize limit. Now what happens? It used to return NULL, now
it gets SIGKILLed. Seriously, about the killing thing, shouldn't we at least
have a timer so two things don't get killed?

 
 ...
 mallocs failed at 64956
 
 Under 4.0-current.
 
 : then one or two processes getting killed.  Having N random processes 
 get
 : malloc() failures can lead to serious instability with processes.
 :
 :Only bad code doesn't check return values of malloc().
 
 You are volunteering to run through the thousands of programs  ports
 to make sure that every one checks the return value for malloc()?
 
 Declarations that don't solve problems are not relevant.  It's bad enough
 that we have to kill something to handle an out-of-vm situation, we
 shouldn't go off and destabilize the rest of the system while we are at 
 it.
 
   -Matt
 
 

I don't tend to use untrustworthy code for production things. I doubt there
are any unchecked mallocs in something as tested as, say, X11R6.3. There could
be..

 Brian Feldman_ __ ___   ___ ___ ___  
 gr...@unixhelp.org_ __ ___ | _ ) __|   \ 
 FreeBSD: The Power to Serve!  _ __ | _ \__ \ |) |
 http://www.freebsd.org   _ |___/___/___/ 



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message



Re: swap-related problems

1999-04-11 Thread Matthew Dillon
: 
: I ran your program.  malloc() appears to work properly -- returns NULL 
when
: the datasize limit is reached.  In my case, I set the datasize limit 
: to 64MB and ran the program.
:
:Unset the datasize limit. Now what happens? It used to return NULL, now
:it gets SIGKILLed. Seriously, about the killing thing, shouldn't we at least
:have a timer so two things don't get killed?

If you unset the datasize limit and the program does not exceed the
maximum system-supported datasize limit, malloc() should not return
NULL even if the system is out of swap.  I don't know why it apparently
did before -- it shouldn't have.   I seem to remember that the default
datasize limit for the system was upped a few months ago.  I don't 
think malloc() operation has changed.

The correct solution is to set a maximum datasize limit.  That is what
it is there for.  I generally unlimit the datasize for root and set it
to around 64MB for all other users.

-Matt
Matthew Dillon 
dil...@backplane.com



To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-current in the body of the message