Re: swap space issues

[Chris]

On Mon, 13 Jul 2020 00:45:36 -0500 Scott Bennett benn...@sdf.org said


Don Wilde  wrote:

>
> On 7/11/20 11:28 PM, Scott Bennett via freebsd-stable wrote:
> >   I have read this entire thread to date with growing dismay, and I
> > thank Donald Wilde for reporting his ongoing troubles, although they
> > spoil my hopes that the kernel's memory management bugs that first became
> > apparent in 11.2-RELEASE (and -STABLE around the same time) were not
> > propagated into 12.x.  A recent update to stable/12 source tree made it
> > finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I
> > was just about to install the upgrade when this thread appeared.
> Spoiler alert. Since I gave up on Synth, I haven't had a single swap 
> issue. It does appear to be one particular port that drove it nuts 
> (apparently, one of the 'Google performance' bits, with a 
> mismatched-brackets problem). I have rebuilt the machine several times, 
> but that's more for my sense of tidiness than anything.

>
> I've got a little Crystal script that walks the installed packages and 
> ports and updates them with system() calls.

> The machine is very slow, but it's not swapping at all.

That's good.  I use portmaster, but not often at present because a
"portmaster -a" run can only be done two or three times per boot before real
memory is locked down to the extent that the system is no longer functional
(i.e., even a scrub of ZFS pools comes to a halt in mid scrub due to lack of
a
sufficient supply of free page frames).
The build procedures of certain ports consistently get killed by the
OOM
killer, along with much collateral damage.  I've noticed that lang/golang
and
lang/rust are prime examples now, although both used to build without
problems.
>
> It is quite usable now with 12-STABLE.

I don't see any good reason to go through the hassle and lost time of
an
upgrade across a major release boundary if I still won't have a production
OS
afterward.  I'm already dealing with a graphics stack rendered unsafe to use
by
the ongoing churn in X11 code.  (See PR #247441, kindly filed for me by Pau
Amma.)
> >
> >   On Fri, 26 Jun 2020 03:55:04 -0700 : Donald Wilde 
> > wrote:
> >
> >> On 6/26/20, Peter Jeremy  wrote:
> >>>
> [snip]
> >>> I strongly suggest you don't have more than one swap device on spinning
> >>> rust - the VM system will stripe I/O across the available devices and
> >>> that will give particularly poor results when it has to seek between the
> >>> partitions.
> >   True.  The only reason I can think of to use more than one swapping/
> > paging area on the same device for the same OS instance is for emergencies
> > or highly unusual, temporary situations in which more space is needed
> until
> > those situations conclude. and even in such situations, if the space can
> be
> > found on another device, it should be placed there.  Interleaving of swap
> > space across multiple devices is intended as a performance enhancement
> > akin to striping (a.k.a. RAID0), although the virtual memory isn't
> > necessarily always actually striped across those devices.  Adding a paging
> > area on the same device as an existing one is an abhorrent situation, as
> > Peter Jeremy noted, and it should be eliminated via swapoff(8) as soon as
> > the extraordinary situation has passed.  N.B. the GENERIC kernel sets a
> > limit of four swap devices, although it can be rebuilt with a different
> > limit.
> That's good data, Scott, thanks! The only reason I got into that 
> situation of trying to add another swap device was that it was crashing 
> with OO swap messages.


I don't recall you posting those messages, but it sounds like exactly
the
*temporary* situation in which adding an inappropriately placed paging area
can
be used long enough to get you out of a bind without a reboot, even though
performance will probably suffer until you have removed it again.  Poor
performance is usually preferable to no performance if it is only temporary.
One cautionary note in such situations, though, applies to remote
paging
areas.  Sparse files allocated on the remote system should not be used as
paging areas.  For example, I discovered the hard way (i.e., the problem was
not documented) that SunOS would crash if a sparse file via NFS were added
as
a paging area and the SunOS system tried to write a page out to an
unallocated
region of the file, which was essentially all of the file at first.

> >> My intent is to make this machine function -- getting the bear
> >> dancing. How deftly she dances is less important than that she dances
> >> at all. My for-real boxen will have real HP and real cores and RAM.
> >>
> >>> Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
> >>> through your boot messages, I expect you'll find messages like:
> >>> warning: total configured swap (524288 pages) exceeds maximum
> recommended
> >>> amount (498848 pages).
> >>> warning: increase kern.maxswzone or 

Re: swap space issues

[Scott Bennett via freebsd-stable]

Don Wilde  wrote:

>
> On 7/11/20 11:28 PM, Scott Bennett via freebsd-stable wrote:
> >   I have read this entire thread to date with growing dismay, and I
> > thank Donald Wilde for reporting his ongoing troubles, although they
> > spoil my hopes that the kernel's memory management bugs that first became
> > apparent in 11.2-RELEASE (and -STABLE around the same time) were not
> > propagated into 12.x.  A recent update to stable/12 source tree made it
> > finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I
> > was just about to install the upgrade when this thread appeared.
> Spoiler alert. Since I gave up on Synth, I haven't had a single swap 
> issue. It does appear to be one particular port that drove it nuts 
> (apparently, one of the 'Google performance' bits, with a 
> mismatched-brackets problem). I have rebuilt the machine several times, 
> but that's more for my sense of tidiness than anything.
>
> I've got a little Crystal script that walks the installed packages and 
> ports and updates them with system() calls.
> The machine is very slow, but it's not swapping at all.

 That's good.  I use portmaster, but not often at present because a
"portmaster -a" run can only be done two or three times per boot before real
memory is locked down to the extent that the system is no longer functional
(i.e., even a scrub of ZFS pools comes to a halt in mid scrub due to lack of a
sufficient supply of free page frames).
 The build procedures of certain ports consistently get killed by the OOM
killer, along with much collateral damage.  I've noticed that lang/golang and
lang/rust are prime examples now, although both used to build without problems.
>
> It is quite usable now with 12-STABLE.

 I don't see any good reason to go through the hassle and lost time of an
upgrade across a major release boundary if I still won't have a production OS
afterward.  I'm already dealing with a graphics stack rendered unsafe to use by
the ongoing churn in X11 code.  (See PR #247441, kindly filed for me by Pau
Amma.)
> >
> >   On Fri, 26 Jun 2020 03:55:04 -0700 : Donald Wilde 
> > wrote:
> >
> >> On 6/26/20, Peter Jeremy  wrote:
> >>>
> [snip]
> >>> I strongly suggest you don't have more than one swap device on spinning
> >>> rust - the VM system will stripe I/O across the available devices and
> >>> that will give particularly poor results when it has to seek between the
> >>> partitions.
> >   True.  The only reason I can think of to use more than one swapping/
> > paging area on the same device for the same OS instance is for emergencies
> > or highly unusual, temporary situations in which more space is needed until
> > those situations conclude. and even in such situations, if the space can be
> > found on another device, it should be placed there.  Interleaving of swap
> > space across multiple devices is intended as a performance enhancement
> > akin to striping (a.k.a. RAID0), although the virtual memory isn't
> > necessarily always actually striped across those devices.  Adding a paging
> > area on the same device as an existing one is an abhorrent situation, as
> > Peter Jeremy noted, and it should be eliminated via swapoff(8) as soon as
> > the extraordinary situation has passed.  N.B. the GENERIC kernel sets a
> > limit of four swap devices, although it can be rebuilt with a different
> > limit.
> That's good data, Scott, thanks! The only reason I got into that 
> situation of trying to add another swap device was that it was crashing 
> with OO swap messages.

 I don't recall you posting those messages, but it sounds like exactly the
*temporary* situation in which adding an inappropriately placed paging area can
be used long enough to get you out of a bind without a reboot, even though
performance will probably suffer until you have removed it again.  Poor
performance is usually preferable to no performance if it is only temporary.
 One cautionary note in such situations, though, applies to remote paging
areas.  Sparse files allocated on the remote system should not be used as
paging areas.  For example, I discovered the hard way (i.e., the problem was
not documented) that SunOS would crash if a sparse file via NFS were added as
a paging area and the SunOS system tried to write a page out to an unallocated
region of the file, which was essentially all of the file at first.

> >> My intent is to make this machine function -- getting the bear
> >> dancing. How deftly she dances is less important than that she dances
> >> at all. My for-real boxen will have real HP and real cores and RAM.
> >>
> >>> Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
> >>> through your boot messages, I expect you'll find messages like:
> >>> warning: total configured swap (524288 pages) exceeds maximum recommended
> >>> amount (498848 pages).
> >>> warning: increase kern.maxswzone or reduce amount of swap.
> >   Also true.  Unfortunately, no guidance whatsoever 

Re: swap space issues

[Don Wilde]

On 7/12/20 12:39 PM, Jonathan Chen wrote:
[snip]

With synth you can reduce the number of workers to just "1" (ie:
Number_of_builders=1), if you just want your ports-build to complete
without any stress. However, one of the reasons why I use synth is
_because_ of the stress it can place on my 12-STABLE snapshots. If the
system is stable and performs well when under load, I feel just that
bit more assured about using it in production environments.

My 2 cents.
Yeah, I did that. Problem was a bad update to a port, had mismatched 
bracket element so blew the stack.


Same thing happened with one worker, one task. Made sure I didn't use 
that port again... ;-)


--
Don Wilde

* What is the Internet of Things but a system  *
* of systems including humans? *


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Jonathan Chen]

On Mon, 13 Jul 2020 at 02:24, Don Wilde  wrote:
> On 7/11/20 11:28 PM, Scott Bennett via freebsd-stable wrote:
> >   I have read this entire thread to date with growing dismay, and I
> > thank Donald Wilde for reporting his ongoing troubles, although they
> > spoil my hopes that the kernel's memory management bugs that first became
> > apparent in 11.2-RELEASE (and -STABLE around the same time) were not
> > propagated into 12.x.  A recent update to stable/12 source tree made it
> > finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I
> > was just about to install the upgrade when this thread appeared.
> Spoiler alert. Since I gave up on Synth, I haven't had a single swap
> issue. It does appear to be one particular port that drove it nuts
> (apparently, one of the 'Google performance' bits, with a
> mismatched-brackets problem). I have rebuilt the machine several times,
> but that's more for my sense of tidiness than anything.

With synth you can reduce the number of workers to just "1" (ie:
Number_of_builders=1), if you just want your ports-build to complete
without any stress. However, one of the reasons why I use synth is
_because_ of the stress it can place on my 12-STABLE snapshots. If the
system is stable and performs well when under load, I feel just that
bit more assured about using it in production environments.

My 2 cents.
-- 
Jonathan Chen 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Don Wilde]

On 7/11/20 11:28 PM, Scott Bennett via freebsd-stable wrote:

  I have read this entire thread to date with growing dismay, and I
thank Donald Wilde for reporting his ongoing troubles, although they
spoil my hopes that the kernel's memory management bugs that first became
apparent in 11.2-RELEASE (and -STABLE around the same time) were not
propagated into 12.x.  A recent update to stable/12 source tree made it
finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I
was just about to install the upgrade when this thread appeared.
Spoiler alert. Since I gave up on Synth, I haven't had a single swap 
issue. It does appear to be one particular port that drove it nuts 
(apparently, one of the 'Google performance' bits, with a 
mismatched-brackets problem). I have rebuilt the machine several times, 
but that's more for my sense of tidiness than anything.


I've got a little Crystal script that walks the installed packages and 
ports and updates them with system() calls.

The machine is very slow, but it's not swapping at all.

It is quite usable now with 12-STABLE.


  On Fri, 26 Jun 2020 03:55:04 -0700 : Donald Wilde 
wrote:


On 6/26/20, Peter Jeremy  wrote:



[snip]

I strongly suggest you don't have more than one swap device on spinning
rust - the VM system will stripe I/O across the available devices and
that will give particularly poor results when it has to seek between the
partitions.

  True.  The only reason I can think of to use more than one swapping/
paging area on the same device for the same OS instance is for emergencies
or highly unusual, temporary situations in which more space is needed until
those situations conclude. and even in such situations, if the space can be
found on another device, it should be placed there.  Interleaving of swap
space across multiple devices is intended as a performance enhancement
akin to striping (a.k.a. RAID0), although the virtual memory isn't
necessarily always actually striped across those devices.  Adding a paging
area on the same device as an existing one is an abhorrent situation, as
Peter Jeremy noted, and it should be eliminated via swapoff(8) as soon as
the extraordinary situation has passed.  N.B. the GENERIC kernel sets a
limit of four swap devices, although it can be rebuilt with a different
limit.
That's good data, Scott, thanks! The only reason I got into that 
situation of trying to add another swap device was that it was crashing 
with OO swap messages.

My intent is to make this machine function -- getting the bear
dancing. How deftly she dances is less important than that she dances
at all. My for-real boxen will have real HP and real cores and RAM.


Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
through your boot messages, I expect you'll find messages like:
warning: total configured swap (524288 pages) exceeds maximum recommended
amount (498848 pages).
warning: increase kern.maxswzone or reduce amount of swap.

  Also true.  Unfortunately, no guidance whatsoever is provided to advise
system administrators who need more space as to how to increase the relevant
table sizes and limits.  However, that is a documentation bug, not a code
bug.
I've got both my kern.max* and CCACHE set up mostly correctly. 
Everything builds and runs well, although I've found that it's helpful 
to only use -j3 while building, not -j4 which would be appropriate for 
my HAMMER i3. I'd much rather have the bear *dancing* than running into 
walls. :D

Yes, as I posted, those were part of the failure stream from the synth
program. When I had kern.maxswzone increased, it got through boot
without complaining.


or maybe:
WARNING: reducing swap size to maximum of MB per unit

The warnings were there, in the as-it-failed complaints.


The absolute limit on swap space is vm.swap_maxpages pages but the
realistic
limit is about half that.  By default the realistic limit is about 4?RAM
(on
64-bit architectures), but this can be adjusted via kern.maxswzone (which
defines the #bytes of RAM to allocate to swzone structures - the actual
space allocated is vm.swzone).

As a further piece of arcana, vm.pageout_oom_seq is a count that controls
the number of passes before the pageout daemon gives up and starts killing
processes when it can't free up enough RAM.  "out of swap space" messages
generally mean that this number is too low, rather than there being a
shortage of swap - particularly if your swap device is rather slow.


Thanks, Peter!

  A second round of thanks to Peter Jeremy for pointing out this sysctl
variable (vm.pageout_oom_seq), although thus far I have yet to see that it is
actually effective in working around the memory management bugs.  I have added
the following lines to /etc/sysctl.conf.

# Because FreeBSD 11.{2,3,4} tie up page frames unnecessarily, set value high
#vm.pageout_wakeup_thresh=14124 # Default value
vm.pageout_wakeup_thresh=112640 # 410 MB


[snip]

I do totally agree that these are 

Re: swap space issues

[Scott Bennett via freebsd-stable]

 I have read this entire thread to date with growing dismay, and I
thank Donald Wilde for reporting his ongoing troubles, although they
spoil my hopes that the kernel's memory management bugs that first became
apparent in 11.2-RELEASE (and -STABLE around the same time) were not
propagated into 12.x.  A recent update to stable/12 source tree made it
finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I
was just about to install the upgrade when this thread appeared.

 On Fri, 26 Jun 2020 03:55:04 -0700 : Donald Wilde 
wrote:

>On 6/26/20, Peter Jeremy  wrote:
>> On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>>>Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>>
>>>Device  1K-blocks UsedAvail Capacity
>>>/dev/ada0s1b 335544320 33554432 0%
>>>/dev/ada0s1d 335544320 33554432 0%
>>>Total671088640 67108864 0%
>>
>> I strongly suggest you don't have more than one swap device on spinning
>> rust - the VM system will stripe I/O across the available devices and
>> that will give particularly poor results when it has to seek between the
>> partitions.

 True.  The only reason I can think of to use more than one swapping/
paging area on the same device for the same OS instance is for emergencies
or highly unusual, temporary situations in which more space is needed until
those situations conclude. and even in such situations, if the space can be
found on another device, it should be placed there.  Interleaving of swap
space across multiple devices is intended as a performance enhancement
akin to striping (a.k.a. RAID0), although the virtual memory isn't
necessarily always actually striped across those devices.  Adding a paging
area on the same device as an existing one is an abhorrent situation, as
Peter Jeremy noted, and it should be eliminated via swapoff(8) as soon as
the extraordinary situation has passed.  N.B. the GENERIC kernel sets a
limit of four swap devices, although it can be rebuilt with a different
limit.
>
>My intent is to make this machine function -- getting the bear
>dancing. How deftly she dances is less important than that she dances
>at all. My for-real boxen will have real HP and real cores and RAM.
>
>>
>> Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
>> through your boot messages, I expect you'll find messages like:
>> warning: total configured swap (524288 pages) exceeds maximum recommended
>> amount (498848 pages).
>> warning: increase kern.maxswzone or reduce amount of swap.

 Also true.  Unfortunately, no guidance whatsoever is provided to advise
system administrators who need more space as to how to increase the relevant
table sizes and limits.  However, that is a documentation bug, not a code
bug.
>
>Yes, as I posted, those were part of the failure stream from the synth
>program. When I had kern.maxswzone increased, it got through boot
>without complaining.
>
>> or maybe:
>> WARNING: reducing swap size to maximum of MB per unit
>
>The warnings were there, in the as-it-failed complaints.
>
>> The absolute limit on swap space is vm.swap_maxpages pages but the
>> realistic
>> limit is about half that.  By default the realistic limit is about 4?RAM
>> (on
>> 64-bit architectures), but this can be adjusted via kern.maxswzone (which
>> defines the #bytes of RAM to allocate to swzone structures - the actual
>> space allocated is vm.swzone).
>>
>> As a further piece of arcana, vm.pageout_oom_seq is a count that controls
>> the number of passes before the pageout daemon gives up and starts killing
>> processes when it can't free up enough RAM.  "out of swap space" messages
>> generally mean that this number is too low, rather than there being a
>> shortage of swap - particularly if your swap device is rather slow.
>>
>Thanks, Peter!

 A second round of thanks to Peter Jeremy for pointing out this sysctl
variable (vm.pageout_oom_seq), although thus far I have yet to see that it is
actually effective in working around the memory management bugs.  I have added
the following lines to /etc/sysctl.conf.

# Because FreeBSD 11.{2,3,4} tie up page frames unnecessarily, set value high
#vm.pageout_wakeup_thresh=14124 # Default value
vm.pageout_wakeup_thresh=112640 # 410 MB

Between the two changes, the pagedaemon *seems* to have stopped killing 
important
processes (or others) for now, which is a huge improvement and relief.  Too bad
FreeBSD needs the changes to be made to keep the system usable somewhat longer.
 My system has 8 GB of real memory.  The kernel apparently refuses to swap
in *any* process, even one as small as /bin/sh, when the free page frame list
has less that ~410 MB of page frames on it.  Setting the 
vm.pageout_wakeup_thresh
to at least 410 MB *seems* to help reduce the number of times a process that
has been marked as swapped out when the system has been under some form of
memory pressure, but it doesn't stop it from happening 

Re: swap space issues

[Peter Jeremy]

On 2020-Jun-28 12:33:21 -0700, Donald Wilde  wrote:
>On 6/28/20, Donald Wilde  wrote:
>> On 6/27/20, Donald Wilde  wrote:
>>> 'spinning rust'  for a disk. My loader.conf has
>>> kern.maxswzone=420 and ccache is fully active and working for both
>>> root on tcsh and users on sh.

Based on my calculations, that maxswzone is good for just under 1GB
swap.  What do you see have for vm.swap_maxpages and vm.swzone?

>> Synth is still crashing hard, same issue.
>An update. Synth still crashed with one swap zone of 16GB.

What do you mean by "swap zone"?  Do you mean you have one 16GB swap
device?

>stack overflow. As I say, there was no warning. Everything was fine,
>then memory usage went through the roof!

I've just tried building llvm80 via ports[1] on my laptop, using the
same options as you.  I have 4GB RAM and 4GB swap with system defaults
and had no problems with an 8-way build.  The highest swap usage I
noticed was <500MB.  I suspect your problems are related to either
ccache or synth.

>The second one, hopefully, contains every log up to the one that
>crashed and hopefully also the beginning of that task. As I say, ONE
>builder and ONE task, after a reboot. LLVM80 was the only builder
>input.

"one builder and one task" - these are presumably synth terms since
they aren't standard ports building terms.  You should be able to
do a single-theaded build of llvm80 in 4GB RAM without problems.

That said, I notice that the first log file suggests you were building
3 ports in parallel, and each port build was running 3 jobs - that's 9
jobs in parallel on a low-spec CPU with 4 threads.  You should limit
the number of CPU-bound processes to the number of CPU threads you have.

[1] cd /usr/ports/devel/llvm80 && make
-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: swap space issues

[Mark Millard via freebsd-stable]

On 2020-Jun-29, at 14:12, Donald Wilde  wrote:

> On 6/29/20, Mark Millard  wrote:
>> [I'm now subscribed so my messages should go through to
>> the list.]
>> 
>> On 2020-Jun-29, at 06:17, Donald Wilde  wrote:
>> 
>>> . . .
>> 
>> You report using:
>> 
>> # For possibly insufficient swap/paging space
>> # (might run out), increase the pageout delay
>> # that leads to Out Of Memory killing of
>> # processes:
>> vm.pfault_oom_attempts= 10
>> vm.pfault_oom_wait= 1
>> # (The multiplication is the total but there
>> # are other potential tradoffs in the factors
>> # multiplied for the same total.)
>> 
>> Note: kib might be interested in what happens
>> for, say, 10 and 1, 5 and 2, and 1 and 10.
>> He has asked for such before from someone
>> having OOM problems but, to my knowledge,
>> no one has taken him up on such testing.
>> (He might be only after 10/1 and 1/10 or
>> other specific figures. Best to ask him if
>> you want to try such things for him.)
> 
> Who is 'kib'? I'm still learning the current team of the Project.

Konstantin Belousov

Also known as kib (from kib at freebsd.org).
Also known as kostik (from part of his gmail address?).


>> I've always set up to use vm.pfault_oom_attempts=-1
>> (avoiding running out of swap space by how I
>> configure things and what I choose to run). I
>> avoid things like tempfs that compete for RAM,
>> especially in low memory contexts.
> 
> Until you explained what you have taught me, I thought these were
> swap-related issues.
> 
> TBH, I am getting disgusted with Synth, as good as it (by spec, not
> actuality) is supposed to be.

While I experimented with Synth a little a long time ago,
I normally stick to tools and techniques that work across
amd64, powerpc64, aarch64, 32-bit powerpc, and armv7 when
I can. So, the experiment was strictly temporary on one
environment at the time.

> CCache I've used for years, and never had this kind of issue.
>> 
>> For 64-bit environments I've never had to have
>> enough swapspace that the boot reported an issue
>> for kern.maxswapzone : more swap is allowed for
>> the same amount of RAM as is allowed for a 32-bit
>> environment.
> 
> Now that you've opened the possibility, it would explain how it goes
> from <3% swap use to OOM in moments... it's not a swap usage issue!
> That's an important thing to learn.
> 
> Not having heard from anyone else, I'm in the process of zeroing my
> drive and starting over.
>> 
>> In the 64-bit type of context with 1 GiByte+
>> of RAM I do -j4 build world buildkernel, 3072 MiBytes
>> of swap. For 2 GiByte+ of RAM I use 4 poudriere builders
>> (one per core), each allowed 4 processes
>> (ALLOW_MAKE_JOBS=yes), so the load average can at times
>> reach around 16 over significant periods. I also use
>> USB SSDs instead of spinning rust. The port builds
>> include a couple of llvm's and other toolchains. But
>> there could be other stuff around that would not fit.
>> 
>> (So synth for you vs. poudriere for me is a
>> difference in our contexts. ALso, I stick to
>> default kern.maxswapzone use without boot
>> messages about exceeding the maximum
>> recommended amount. Increasing kern.maxswapzone
>> trades off KVM available for other purposes and
>> I avoid the tradeoffs that I do not understand.)
> [snip]
>> (My context is head, not stable.)
> 
> Thanks for documenting your usage. I'll store a pointer to this week's
> -stable archives  so I can come back to this when I get to smaller
> builds.
>> 
>> . . .
>> 
>>> What got corrupted was one of the /usr/.ccache directories, but
>>> 'ccache -C' doesn't clear it.
>> 
>> I've not used ccache. So that is another variation
>> in our contexts.
>> 
>> I use UFS, not ZFS. I avoid tmpfs and such that complete
>> for memory.
> 
> I'm using UFS on MBR partitions.

GPT for root file systems for me, other than any old PowerMacs
(APM). (On the small arm's I just use microsd cards to get to
booting the root file system on a GPT based USB SSD via a
technique that works the same for all such arms that I
sometimes have access to, other than the RPi4's at this stage.)

>> . . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/29/20, Mark Millard  wrote:
> [I'm now subscribed so my messages should go through to
> the list.]
>
> On 2020-Jun-29, at 06:17, Donald Wilde  wrote:
>
>> [adding maintainers of synth and ccache]
>>
>> On 6/29/20, Mark Millard  wrote:
>>> Based on "small arm system" context experiments
>>> mostly . . .
>>>
>>> If your console messasges do not include
>>> messages about "swap_pager_getswapspace(...): failed",
>>> then it is unlikely that being out of swap space
>>> is the actual issue even when it reports: "was killed:
>>> out of swap space" messages. For such contexts, making
>>> the swap area bigger does not help.
>>>
>>
>> It did not show those getswapspace messages.
>
> Any other potentially of interest console messages?
>
>>> In other words, "was killed: out of swap space"
>>> is frequently a misnomer and not to be believed
>>> for "why" the kill happened or what should be
>>> done about it --without other evidence also being
>>> present anyway.
>>>
>>> Other causes include:
>>>
>>> Sustained low free RAM (via stays-runnable processes).
>>> A sufficiently delayed pageout.
>>> The swap blk uma zone was exhausted.
>>> The swap pctrie uma zone was exhausted.
>>>
>>> (stays-runnable processes are not swapped out
>>> [kernel stacks are not swapped out] but do actively
>>> compete for RAM via paging activity. In such a
>>> context, free RAM can stay low.)
>>>
>>> The below material does not deal with the
>>> the "exhausted" causes but does deal with
>>> the other 2.
>>>
>>> Presuming that you are getting "was killed: out
>>> of swap space" notices but are not getting
>>> "swap_pager_getswapspace failed" notices and
>>> that kern.maxswzone vs. system load has not
>>> been adjusted in a way that leads to bad
>>> memory tradeoffs . . .
>>>
>>> I recommend attempting use of, say, (from
>>> my /etc/sysctl.conf ):
>>>
>> Attached is what I tried, but when I ran synth again, I got a
>> corrupted HDD that fsck refuses to fix, whether in 1U mode or with fs
>> mounted. It just will not SALVAGE even when I add the -y flag.
>
> That is a horrible result.
>
> I assume that you rebooted after editing

Yes.

> sysctl.conf or manually applied the
> values separately instead.
>
> What sort of console messages were generated?
> Was the corruption the only issue? Did the system
> crash? In what way?

The symptoms used to be that, once synth crashed by OOM, it would
refuse to allow any packages to be added, because the synth data
structures were blasted. Synth is unfortunately pretty blockheaded in
that you can't control the order of the ports built. You can control
how many ports get built at a time and how many tasks per port get
started and managed. As I say, though, this change led to -- I won't
say _caused_ -- a corruption of the disk area used for ccache. I could
not recover from that, although I could have turned ccache off.

I was able to manually build llvm80 from the ports, so the dependency
gotcha that gave me the issues was in some other dependency. As I said
in my other message, it seems to be the Google performance tools or
Brotli that might have been broken.

What's great, Mark, is that you've given me a lot more insight into
the tunability of the kernel. As I said, I do a lot of IoT and
embedded work with ARM-based Micro-Controllers.
>
> Your notes on what you set have a incorrect
> comment about a case that you did not use:
>
> # For plunty of swap/paging space (will not
> # run out), avoid pageout delays leading to
> # Out Of Memory killing of processes:
> #vm.pfault_oom_attempts=-1 # infinite
>
> vm.pfault_oom_attempts being -1 is a special
> value that disables the the logic for the
> vm.pfault_oom_attempts and vm.pfault_oom_wait
> pair: Willing to wait indefinitely relative to
> how long the pageout takes, no retries. (Other
> OOM criteria may still be active.)

Ah, I appreciate the distinction. :)
>
> You report using:
>
> # For possibly insufficient swap/paging space
> # (might run out), increase the pageout delay
> # that leads to Out Of Memory killing of
> # processes:
> vm.pfault_oom_attempts= 10
> vm.pfault_oom_wait= 1
> # (The multiplication is the total but there
> # are other potential tradoffs in the factors
> # multiplied for the same total.)
>
> Note: kib might be interested in what happens
> for, say, 10 and 1, 5 and 2, and 1 and 10.
> He has asked for such before from someone
> having OOM problems but, to my knowledge,
> no one has taken him up on such testing.
> (He might be only after 10/1 and 1/10 or
> other specific figures. Best to ask him if
> you want to try such things for him.)

Who is 'kib'? I'm still learning the current team of the Project.
>
> I've always set up to use vm.pfault_oom_attempts=-1
> (avoiding running out of swap space by how I
> configure things and what I choose to run). I
> avoid things like tempfs that compete for RAM,
> especially in low memory contexts.

Until you explained what you have taught me, I thought these were
swap-related issues.

TBH, I am 

Re: swap space issues

[Mark Millard via freebsd-stable]

[I'm now subscribed so my messages should go through to
the list.]

On 2020-Jun-29, at 06:17, Donald Wilde  wrote:

> [adding maintainers of synth and ccache]
> 
> On 6/29/20, Mark Millard  wrote:
>> Based on "small arm system" context experiments
>> mostly . . .
>> 
>> If your console messasges do not include
>> messages about "swap_pager_getswapspace(...): failed",
>> then it is unlikely that being out of swap space
>> is the actual issue even when it reports: "was killed:
>> out of swap space" messages. For such contexts, making
>> the swap area bigger does not help.
>> 
> 
> It did not show those getswapspace messages.

Any other potentially of interest console messages?

>> In other words, "was killed: out of swap space"
>> is frequently a misnomer and not to be believed
>> for "why" the kill happened or what should be
>> done about it --without other evidence also being
>> present anyway.
>> 
>> Other causes include:
>> 
>> Sustained low free RAM (via stays-runnable processes).
>> A sufficiently delayed pageout.
>> The swap blk uma zone was exhausted.
>> The swap pctrie uma zone was exhausted.
>> 
>> (stays-runnable processes are not swapped out
>> [kernel stacks are not swapped out] but do actively
>> compete for RAM via paging activity. In such a
>> context, free RAM can stay low.)
>> 
>> The below material does not deal with the
>> the "exhausted" causes but does deal with
>> the other 2.
>> 
>> Presuming that you are getting "was killed: out
>> of swap space" notices but are not getting
>> "swap_pager_getswapspace failed" notices and
>> that kern.maxswzone vs. system load has not
>> been adjusted in a way that leads to bad
>> memory tradeoffs . . .
>> 
>> I recommend attempting use of, say, (from
>> my /etc/sysctl.conf ):
>> 
> Attached is what I tried, but when I ran synth again, I got a
> corrupted HDD that fsck refuses to fix, whether in 1U mode or with fs
> mounted. It just will not SALVAGE even when I add the -y flag.

That is a horrible result.

I assume that you rebooted after editing
sysctl.conf or manually applied the
values separately instead.

What sort of console messages were generated?
Was the corruption the only issue? Did the system
crash? In what way?

Your notes on what you set have a incorrect
comment about a case that you did not use:

# For plunty of swap/paging space (will not
# run out), avoid pageout delays leading to
# Out Of Memory killing of processes:
#vm.pfault_oom_attempts=-1 # infinite

vm.pfault_oom_attempts being -1 is a special
value that disables the the logic for the
vm.pfault_oom_attempts and vm.pfault_oom_wait
pair: Willing to wait indefinitely relative to
how long the pageout takes, no retries. (Other
OOM criteria may still be active.)

You report using:

# For possibly insufficient swap/paging space
# (might run out), increase the pageout delay
# that leads to Out Of Memory killing of
# processes:
vm.pfault_oom_attempts= 10
vm.pfault_oom_wait= 1
# (The multiplication is the total but there
# are other potential tradoffs in the factors
# multiplied for the same total.)

Note: kib might be interested in what happens
for, say, 10 and 1, 5 and 2, and 1 and 10.
He has asked for such before from someone
having OOM problems but, to my knowledge,
no one has taken him up on such testing.
(He might be only after 10/1 and 1/10 or
other specific figures. Best to ask him if
you want to try such things for him.)

I've always set up to use vm.pfault_oom_attempts=-1
(avoiding running out of swap space by how I
configure things and what I choose to run). I
avoid things like tempfs that compete for RAM,
especially in low memory contexts.

For 64-bit environments I've never had to have
enough swapspace that the boot reported an issue
for kern.maxswapzone : more swap is allowed for
the same amount of RAM as is allowed for a 32-bit
environment.

In the 64-bit type of context with 1 GiByte+
of RAM I do -j4 build world buildkernel, 3072 MiBytes
of swap. For 2 GiByte+ of RAM I use 4 poudriere builders
(one per core), each allowed 4 processes
(ALLOW_MAKE_JOBS=yes), so the load average can at times
reach around 16 over significant periods. I also use
USB SSDs instead of spinning rust. The port builds
include a couple of llvm's and other toolchains. But
there could be other stuff around that would not fit.

(So synth for you vs. poudriere for me is a
difference in our contexts. ALso, I stick to
default kern.maxswapzone use without boot
messages about exceeding the maximum
recommended amount. Increasing kern.maxswapzone
trades off KVM available for other purposes and
I avoid the tradeoffs that I do not understand.)

For 32-bit environments with environments with
2 GiByte+ of RAM I have to be more careful to
be sure of avoiding running out of swap for
what I do. PARALLEL_JOBS=2 and ALLOW_MAKE_JOBS=yes
for poudruere (so load average around 8 over some
periods). -j4 for buildworld buildkernel .

For 32-bit 1 GiByte I used -j2 for buildworld
buildkernel , 1800 MiBytes of swap. As I 

Re: swap space issues

[Donald Wilde]

[adding maintainers of synth and ccache]

On 6/29/20, Mark Millard  wrote:
> Based on "small arm system" context experiments
> mostly . . .
>
> If your console messasges do not include
> messages about "swap_pager_getswapspace(...): failed",
> then it is unlikely that being out of swap space
> is the actual issue even when it reports: "was killed:
> out of swap space" messages. For such contexts, making
> the swap area bigger does not help.
>

It did not show those getswapspace messages.

> In other words, "was killed: out of swap space"
> is frequently a misnomer and not to be believed
> for "why" the kill happened or what should be
> done about it --without other evidence also being
> present anyway.
>
> Other causes include:
>
> Sustained low free RAM (via stays-runnable processes).
> A sufficiently delayed pageout.
> The swap blk uma zone was exhausted.
> The swap pctrie uma zone was exhausted.
>
> (stays-runnable processes are not swapped out
> [kernel stacks are not swapped out] but do actively
> compete for RAM via paging activity. In such a
> context, free RAM can stay low.)
>
> The below material does not deal with the
> the "exhausted" causes but does deal with
> the other 2.
>
> Presuming that you are getting "was killed: out
> of swap space" notices but are not getting
> "swap_pager_getswapspace failed" notices and
> that kern.maxswzone vs. system load has not
> been adjusted in a way that leads to bad
> memory tradeoffs . . .
>
> I recommend attempting use of, say, (from
> my /etc/sysctl.conf ):
>
Attached is what I tried, but when I ran synth again, I got a
corrupted HDD that fsck refuses to fix, whether in 1U mode or with fs
mounted. It just will not SALVAGE even when I add the -y flag.

What got corrupted was one of the /usr/.ccache directories, but
'ccache -C' doesn't clear it.

I restored the original /etc/sysctl.conf, but I can't add packages or
ports any more, so I'm afraid I'm going to have to dd if=/dev/zero the
disk and reload from 12.1R and start over again.

I can't even 'rm -Rf /usr/.ccache'. It says 'Directory not empty'.

I don't need this system up and running, so I'm not going to make any
more changes until I see if any of you have suggestions to try first.
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *



sysctl.conf.new
Description: Binary data
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/28/20, Donald Wilde  wrote:
> Okay, the post with the attachments was too big, so I canceled it. The
> direct copies went out with the archives.
>
> Here's the one pertinent log file. I'm still examining it.
> [snip]

Even the log file is too big. Sent the whole thing to the LLVM80
maintainer and the Synth maintainer.

>> I'll keep examining the entire /var/log/synth archive, which I
>> attached to my last post in this thread. :D
>
> This one will again be bounced by -stable, but hopefully Eric, the
> Synth maintainer will see it.
>
> ... and Brooks (Jason?), the llvm80 maintainer. Will repost to -stable
> without attachments, and with more  analysis.
[snip]

 From the log file for devel/llvm80:

[4911/4933] : && /usr/local/bin/cmake -E rm -f lib/libgtest_main.a &&
/usr/local/bin/ar qc lib/libgtest_main.a
utils/unittest/UnitTestMain/CMakeFiles/gtest_main.dir/TestMain.cpp.o
&& /usr/local/bin/ranlib lib/libgtest_main.a && :
[4912/4933] cd /construction/xports/devel/llvm80/work/.build/docs &&
/usr/local/bin/cmake -E make_directory
/construction/xports/devel/llvm80/work/.build/docs/man &&
/usr/local/bin/sphinx-build-3.7 -b man -d
/construction/xports/devel/llvm80/work/.build/docs/_doctrees-dsymutil-man
-q /construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs
/construction/xports/devel/llvm80/work/.build/docs/man
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:96:
WARNING: Unexpected indentation.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:94:
WARNING: Inline literal start-string without end-string.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:97:
WARNING: Block quote ends without a blank line; unexpected unindent.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:104:
WARNING: Unexpected indentation.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:102:
WARNING: Inline literal start-string without end-string.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:105:
WARNING: Block quote ends without a blank line; unexpected unindent.

[snip]
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:141:
WARNING: Block quote ends without a blank line; unexpected unindent.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/MarkdownQuickstartTemplate.md:145:
WARNING: Unexpected indentation.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/SpeculativeLoadHardening.md:32:
WARNING: Unexpected indentation.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/SpeculativeLoadHardening.md:21:
WARNING: Inline literal start-string without end-string.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/SpeculativeLoadHardening.md:34:
WARNING: Block quote ends without a blank line; unexpected unindent.
/construction/xports/devel/llvm80/work/llvm-8.0.1.src/docs/SpeculativeLoadHardening.md:39:
WARNING: Unexpected indentation.

These go on and on. I'll have to go back and examine the config.

These reference tests of something by Google. IIRC, I enabled BROTLI
and another Google package, something like 'Google Performance Tools.'


-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

[trimming the direct cc:s]

On 6/28/20, Donald Wilde  wrote:
> On 6/28/20, Donald Wilde  wrote:
>> On 6/27/20, Donald Wilde  wrote:
>>> On 6/26/20, Ronald Klop  wrote:

> [snip]
[snip]

I should have said that this run was done with 3 and 3 in synth.

> The first attached tarball was an earlier run that failed (not
> crashed) on cmake. I resolved that one through a direct build, but I
> did do a graceful stoppage of synth while llvm80 was in process. It
> did finally crash anyway, which led me to throttle back to 1 and 1.
>
> The second one, hopefully, contains every log up to the one that
> crashed and hopefully also the beginning of that task. As I say, ONE
> builder and ONE task, after a reboot. LLVM80 was the only builder
> input.

Okay, here's the summary, updated just before the crash. Attached is
the full JSON, but I'll give the highlights here:
{
[snip]
  ,"elapsed":"04:43:47"
... almost five hours

[snip]
  ,"swapinfo":" 1.6%"
... no warning!

  ,"load":" 1.12"
... 1 builder and 1 task
 }
 ,"builders":[
  {
[snip]
   ,"origin":"devel/llvm80"
   ,"lines":"15326"

FWIW, that's a lot of "lines," compared to other ports. What does
"lines" count?
Will look it up.
  }
 ]
}
I'll keep examining the entire /var/log/synth archive, which I
attached to my last post in this thread. :D
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/27/20, Donald Wilde  wrote:
> On 6/26/20, Ronald Klop  wrote:
>>
>> Van: Bob Bishop 
>> Datum: vrijdag, 26 juni 2020 17:18
>> Aan: Peter Jeremy 
>> CC: Donald Wilde , freebsd-stable
>> 
>> Onderwerp: Re: swap space issues
>>>
>>>
>>>
>>> > On 26 Jun 2020, at 11:23, Peter Jeremy  wrote:
>>> >
>>> > On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>>> >> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>> >>
>>> >> Device  1K-blocks UsedAvail Capacity
>>> >> /dev/ada0s1b 335544320 33554432 0%
>>> >> /dev/ada0s1d 335544320 33554432 0%
>>> >> Total671088640 67108864 0%
>>> >
>>> > I strongly suggest you don't have more than one swap device on
>>> > spinning
>>> > rust - the VM system will stripe I/O across the available devices and
>>> > that will give particularly poor results when it has to seek between
>>> > the
>>> > partitions.
>>>
> Based on all recommendations on this thread (thanks, guys!), I've
> rebuilt my i3 mule with exactly one 16G partition, as it has only
> 'spinning rust'  for a disk. My loader.conf has
> kern.maxswzone=420 and ccache is fully active and working for both
> root on tcsh and users on sh.

That appears to be successful. this is still a MBR-based system, not
GPT, due to BIOS issues, but it's a 'choose my battles' situation.

>
> I have yet to try synth again. I'm doing buildworld/buildkernel for
> 12-STABLE, but evidence so far is good. 'top -t' is actually happy,
> showing 16G (grog?), so I'll try firing up synth after another hour or
> so on the latest fetch of the ports tree.

Top appears to have gone south again. I give up, Greg! 'pstat -s' _is_ happy.

Synth is still crashing hard, same issue. As I said in freebsd-ports
on my thread about 'stable postgresql11', pg11 didn't like synth's
request that I create a different directory because
/usr/ports/distfiles didn't exist. Doing so and adding the DISTFILES
to make.conf caused all kinds of problems.

I am currently (I hope) building a successful pg11-server, but I've
restored /usr/ports/distfiles and before I go to bed tonight I hope to
kick off another synth run.

This failure happened with three builders and three tasks per builder.
The symptom, multiple times, has been that llvm80 takes forever and
then causes the swap failure even though pstat -s is happy AND loader
conf kern.maxswzone (=420) is sufficient for my 16G partition.
I've tried installing llvm80 as a pkg, but that didn't help. The swap
stays at zero usage, and then goes almost instantaneously to OOM.

The only bit of advice I'm not completely compliant with is the use of
a 16G swap partition where I was advised to set max to 8G.
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/26/20, Ronald Klop  wrote:
>
> Van: Bob Bishop 
> Datum: vrijdag, 26 juni 2020 17:18
> Aan: Peter Jeremy 
> CC: Donald Wilde , freebsd-stable
> 
> Onderwerp: Re: swap space issues
>>
>>
>>
>> > On 26 Jun 2020, at 11:23, Peter Jeremy  wrote:
>> >
>> > On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>> >> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>> >>
>> >> Device  1K-blocks UsedAvail Capacity
>> >> /dev/ada0s1b 335544320 33554432 0%
>> >> /dev/ada0s1d 335544320 33554432 0%
>> >> Total671088640 67108864 0%
>> >
>> > I strongly suggest you don't have more than one swap device on spinning
>> > rust - the VM system will stripe I/O across the available devices and
>> > that will give particularly poor results when it has to seek between
>> > the
>> > partitions.
>>
Based on all recommendations on this thread (thanks, guys!), I've
rebuilt my i3 mule with exactly one 16G partition, as it has only
'spinning rust'  for a disk. My loader.conf has
kern.maxswzone=420 and ccache is fully active and working for both
root on tcsh and users on sh.

I have yet to try synth again. I'm doing buildworld/buildkernel for
12-STABLE, but evidence so far is good. 'top -t' is actually happy,
showing 16G (grog?), so I'll try firing up synth after another hour or
so on the latest fetch of the ports tree.

Happy coder, me! :D
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

# DeviceMountpoint  FStype  Options DumpPass#
/dev/ada0s1a/   ufs rw  1   1
/dev/ada0s1bnoneswapsw  0   0
/dev/ada0s1d/expufs rw  2   2
fdesc   /dev/fd fdescfs rw  0   0
proc/proc   procfs  rw  0   0

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Ronald Klop]

Van: Bob Bishop 
Datum: vrijdag, 26 juni 2020 17:18
Aan: Peter Jeremy 
CC: Donald Wilde , freebsd-stable 

Onderwerp: Re: swap space issues




> On 26 Jun 2020, at 11:23, Peter Jeremy  wrote:
>
> On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>
>> Device  1K-blocks UsedAvail Capacity
>> /dev/ada0s1b 335544320 33554432 0%
>> /dev/ada0s1d 335544320 33554432 0%
>> Total671088640 67108864 0%
>
> I strongly suggest you don't have more than one swap device on spinning
> rust - the VM system will stripe I/O across the available devices and
> that will give particularly poor results when it has to seek between the
> partitions.
 
If you configure a ZFS mirror in bsdinstall you get a swap partition per drive by default.


If you are running on multiple disks (a mirror) it can provide extra speed. The 
example above is on the same disk. On one disk multiple swap partitions will 
only spread the data non-optimal for the heads of the disk.


> 

> Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
> through your boot messages, I expect you'll find messages like:
> warning: total configured swap (524288 pages) exceeds maximum recommended 
amount (498848 pages).
> warning: increase kern.maxswzone or reduce amount of swap.
> or maybe:
> WARNING: reducing swap size to maximum of MB per unit
>
> The absolute limit on swap space is vm.swap_maxpages pages but the realistic
> limit is about half that.  By default the realistic limit is about 4×RAM (on
> 64-bit architectures), but this can be adjusted via kern.maxswzone (which
> defines the #bytes of RAM to allocate to swzone structures - the actual
> space allocated is vm.swzone).
>
> As a further piece of arcana, vm.pageout_oom_seq is a count that controls
> the number of passes before the pageout daemon gives up and starts killing
> processes when it can't free up enough RAM.  "out of swap space" messages
> generally mean that this number is too low, rather than there being a
> shortage of swap - particularly if your swap device is rather slow.
>
> --
> Peter Jeremy


--
Bob Bishop   t: +44 (0)118 940 1243
r...@gid.co.uk m: +44 (0)783 626 4518




 



 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Bob Bishop]

> On 26 Jun 2020, at 11:23, Peter Jeremy  wrote:
> 
> On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>> 
>> Device  1K-blocks UsedAvail Capacity
>> /dev/ada0s1b 335544320 33554432 0%
>> /dev/ada0s1d 335544320 33554432 0%
>> Total671088640 67108864 0%
> 
> I strongly suggest you don't have more than one swap device on spinning
> rust - the VM system will stripe I/O across the available devices and
> that will give particularly poor results when it has to seek between the
> partitions.

If you configure a ZFS mirror in bsdinstall you get a swap partition per drive 
by default.

> Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
> through your boot messages, I expect you'll find messages like:
> warning: total configured swap (524288 pages) exceeds maximum recommended 
> amount (498848 pages).
> warning: increase kern.maxswzone or reduce amount of swap.
> or maybe:
> WARNING: reducing swap size to maximum of MB per unit
> 
> The absolute limit on swap space is vm.swap_maxpages pages but the realistic
> limit is about half that.  By default the realistic limit is about 4×RAM (on
> 64-bit architectures), but this can be adjusted via kern.maxswzone (which
> defines the #bytes of RAM to allocate to swzone structures - the actual
> space allocated is vm.swzone).
> 
> As a further piece of arcana, vm.pageout_oom_seq is a count that controls
> the number of passes before the pageout daemon gives up and starts killing
> processes when it can't free up enough RAM.  "out of swap space" messages
> generally mean that this number is too low, rather than there being a
> shortage of swap - particularly if your swap device is rather slow.
> 
> --
> Peter Jeremy


--
Bob Bishop   t: +44 (0)118 940 1243
r...@gid.co.uk m: +44 (0)783 626 4518







signature.asc
Description: Message signed with OpenPGP

Re: swap space issues

[Paul Mather]

On Jun 26, 2020, at 6:58 AM, Stefan Eßer  wrote:

> Am 26.06.20 um 12:23 schrieb Peter Jeremy:
>> On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>>> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>> 
>>> Device  1K-blocks UsedAvail Capacity
>>> /dev/ada0s1b 335544320 33554432 0%
>>> /dev/ada0s1d 335544320 33554432 0%
>>> Total671088640 67108864 0%
>> 
>> I strongly suggest you don't have more than one swap device on spinning
>> rust - the VM system will stripe I/O across the available devices and
>> that will give particularly poor results when it has to seek between the
>> partitions.
> 
[[...]]
>> As a further piece of arcana, vm.pageout_oom_seq is a count that controls
>> the number of passes before the pageout daemon gives up and starts killing
>> processes when it can't free up enough RAM.  "out of swap space" messages
>> generally mean that this number is too low, rather than there being a
>> shortage of swap - particularly if your swap device is rather slow.
> 
> I'm not sure that this specific sysctl is documented in such a way
> that it is easy to find by people suffering from out-of-memory kills.
> 
> Perhaps it could be mentioned as a parameter that may need tuning in
> the OOM message?
> 
> And while it does not come up that often in the mail list, it might
> be better for many kinds of application if the default was increased
> (a longer wait for resources might be more acceptable than the loss
> of all results of a long running computation).


The OOM issue is more pressing on platforms like FreeBSD/arm that tend to have 
low RAM and slow writable storage such as SD card.  There have been several 
threads on the issues this creates (e.g., 
https://docs.freebsd.org/cgi/getmsg.cgi?fetch=228789+0+archive/2018/freebsd-arm/20180819.freebsd-arm
 
)
 that have led to some insight into how to tune the OOM killer.  One thing that 
becomes clear is that the "Out of swap space" error message is misleading as 
often it really means "Couldn't obtain RAM in a timely fashion."  On hardware 
such as the Raspberry Pi, it's often the case that the system has enough swap 
space: it's just that it can't write to swap on SD card before the default 
vm.pageout_oom_seq passes are exhausted and so the OOM killer starts reaping 
active processes (like the clang trying to build clang:), and all sorts of 
things start to break. :-)

Cheers,

Paul.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/26/20, Stefan Eßer  wrote:
> Am 26.06.20 um 12:23 schrieb Peter Jeremy:
>> On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>>> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
[snip]
> An idea for a better strategy:
>
> It might be better to use an allocation algorithm that assigns a
> swap device to each running process that needs pages written to the
> swap device and only assign another swap device (and use if from
> then on for that process) if there is no free space left on the one
> used until then.
>
> Such a strategy would at least reduce the number of processes that
> need all configured swap devices at the same time in a striped
> configuration.
>
> If all processes start with the first configured swap device assigned
> to them, this will lead to only one of them being used until it fills
> up, then progressing to the next one.
>
> The strategy of whether the initial swap device assigned to a process
> is always the first one configured in the system, or whether after
> that could not be used by some process is moved on to the next one
> (typically the one assigned to that process for further page-outs) is
> not obvious to me.

You're getting over my head, STefan, but that's okay. I suspect that
having somebody be loony -- and desperate enough -- to configure two
swap partitions is a rare occurance.
>
> The behavior could be controlled by a sysctl to allow to adapt the
> strategy to the hardware (e.g. rotating vs. flash disks for swap).

Not to mention Intel and Micron and their fancy fast non-volatile
chips ('Optane'). I do agree that SOMEBODY is going to need this kind
of sysctl guidance for the kernel.
>
[snip]

> And while it does not come up that often in the mail list, it might
> be better for many kinds of application if the default was increased
> (a longer wait for resources might be more acceptable than the loss
> of all results of a long running computation).

Yes. Synth seems to be able to keep going / recover from last-known
success points, but your point is very valid. As we go further into
OOP, the _controllable_ use of heap space, stack space, and recursion
is going to become more crucial. We humans are used to operating
without a "full stack," (sleep DOES help :) ) but I think the whole
point of modern AI is to create systems that actually can master the
logical inference chains proposed by the early LISP guys at MIT, C-M,
and Stanford. The gains from ML have been enough to keep 'the Street'
happy for now but they'll want more soon enough.

> Regards, STefan

Thought-provoking indeed. :D

-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/25/20, Greg 'groggy' Lehey  wrote:
> On Thursday, 25 June 2020 at 19:31:34 -0700, Donald Wilde wrote:
>> On 6/25/20, Greg 'groggy' Lehey  wrote:
>>> On Wednesday, 24 June 2020 at 23:27:27 -0700, Kevin Oberman wrote,
>>> without trimming:
>>>
 On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
 wrote:
>>>
 gpart(8) works just fine on MBR drives and partitions/slices and
 has a much friendlier user interface. "gpart resize" is the
 command you want.
>>>
>>> Thanks.  I try to offer suggestions that I've tried, and offer an
>>> example.  I haven't tried 'gpart resize', but it looks much easier.
>>
>> 'gpart resize' did work well,
>
> Yes, I saw that from the gpart output you posted.
>
>> although the man page for gpart assumes way too much. I was able to
>> successfully work my way through and create ('gpart add') and mount
>> not just one but two 32G swap partitions.
>
> Yes, I saw that too.  Not quite what I was suggesting: I suspected
> some overflow issue, so the partitions should really have been a
> little shy of 32 GB.  And at least for the start you should only mount
> one of them.  In the unlikely event that it should threaten to fill
> up, you can still mount the other one without rebooting (swapon(1)).

I got greedy! :) i also wanted to embed my newfound understanding of
gpart, geom, and swapping into the noggin so I could move on.
>
> How are things looking now?

So far, it works, but there haven't been enough changes in the ports
tree that synth even needs a cuppa.

I think what I am going to do is to wipe the machine one more time
from 12.1R (with 16G swap, as both you and Peter suggest), STABLE-ize
it, and see if synth can handle the entire installed ports tree
(around 300 primary+dependency ports) without crashing and with
conservative builder-subtask limits. It will take it three days to do
that, but I think I know enough of the gotchas to get it to succeed.

It's a good idea to have a second one in reserve, but when it broke
through the roof it happened way too fast for me to actually enable
anything. I'll see if I can make a one-character script invocation and
try it, though!

>
> Greg

Again, my thanks. You guys are the best! :D

-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Stefan Eßer]

Am 26.06.20 um 12:23 schrieb Peter Jeremy:
> On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>
>> Device  1K-blocks UsedAvail Capacity
>> /dev/ada0s1b 335544320 33554432 0%
>> /dev/ada0s1d 335544320 33554432 0%
>> Total671088640 67108864 0%
> 
> I strongly suggest you don't have more than one swap device on spinning
> rust - the VM system will stripe I/O across the available devices and
> that will give particularly poor results when it has to seek between the
> partitions.

This used to be beneficial, when disk read and write bandwidth was
limited and whole processes had to be swapped in or out due to RAM
pressure. (This changed due to more RAM and a different ratio of
seek to transfer times for a given amount of data.)

An idea for a better strategy:

It might be better to use an allocation algorithm that assigns a
swap device to each running process that needs pages written to the
swap device and only assign another swap device (and use if from
then on for that process) if there is no free space left on the one
used until then.

Such a strategy would at least reduce the number of processes that
need all configured swap devices at the same time in a striped
configuration.

If all processes start with the first configured swap device assigned
to them, this will lead to only one of them being used until it fills
up, then progressing to the next one.

The strategy of whether the initial swap device assigned to a process
is always the first one configured in the system, or whether after
that could not be used by some process is moved on to the next one
(typically the one assigned to that process for further page-outs) is
not obvious to me.

The behavior could be controlled by a sysctl to allow to adapt the
strategy to the hardware (e.g. rotating vs. flash disks for swap).

> As a further piece of arcana, vm.pageout_oom_seq is a count that controls
> the number of passes before the pageout daemon gives up and starts killing
> processes when it can't free up enough RAM.  "out of swap space" messages
> generally mean that this number is too low, rather than there being a
> shortage of swap - particularly if your swap device is rather slow.

I'm not sure that this specific sysctl is documented in such a way
that it is easy to find by people suffering from out-of-memory kills.

Perhaps it could be mentioned as a parameter that may need tuning in
the OOM message?

And while it does not come up that often in the mail list, it might
be better for many kinds of application if the default was increased
(a longer wait for resources might be more acceptable than the loss
of all results of a long running computation).

Regards, STefan
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/26/20, Peter Jeremy  wrote:
> On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>>Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>
>>Device  1K-blocks UsedAvail Capacity
>>/dev/ada0s1b 335544320 33554432 0%
>>/dev/ada0s1d 335544320 33554432 0%
>>Total671088640 67108864 0%
>
> I strongly suggest you don't have more than one swap device on spinning
> rust - the VM system will stripe I/O across the available devices and
> that will give particularly poor results when it has to seek between the
> partitions.

My intent is to make this machine function -- getting the bear
dancing. How deftly she dances is less important than that she dances
at all. My for-real boxen will have real HP and real cores and RAM.

>
> Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
> through your boot messages, I expect you'll find messages like:
> warning: total configured swap (524288 pages) exceeds maximum recommended
> amount (498848 pages).
> warning: increase kern.maxswzone or reduce amount of swap.

Yes, as I posted, those were part of the failure stream from the synth
program. When I had kern.maxswzone increased, it got through boot
without complaining.

> or maybe:
> WARNING: reducing swap size to maximum of MB per unit

The warnings were there, in the as-it-failed complaints.

> The absolute limit on swap space is vm.swap_maxpages pages but the
> realistic
> limit is about half that.  By default the realistic limit is about 4×RAM
> (on
> 64-bit architectures), but this can be adjusted via kern.maxswzone (which
> defines the #bytes of RAM to allocate to swzone structures - the actual
> space allocated is vm.swzone).
>
> As a further piece of arcana, vm.pageout_oom_seq is a count that controls
> the number of passes before the pageout daemon gives up and starts killing
> processes when it can't free up enough RAM.  "out of swap space" messages
> generally mean that this number is too low, rather than there being a
> shortage of swap - particularly if your swap device is rather slow.
>
Thanks, Peter!
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Peter Jeremy]

On 2020-Jun-25 11:30:31 -0700, Donald Wilde  wrote:
>Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>
>Device  1K-blocks UsedAvail Capacity
>/dev/ada0s1b 335544320 33554432 0%
>/dev/ada0s1d 335544320 33554432 0%
>Total671088640 67108864 0%

I strongly suggest you don't have more than one swap device on spinning
rust - the VM system will stripe I/O across the available devices and
that will give particularly poor results when it has to seek between the
partitions.

Also, you can't actually use 64GB swap with 4GB RAM.  If you look back
through your boot messages, I expect you'll find messages like:
warning: total configured swap (524288 pages) exceeds maximum recommended 
amount (498848 pages).
warning: increase kern.maxswzone or reduce amount of swap.
or maybe:
WARNING: reducing swap size to maximum of MB per unit

The absolute limit on swap space is vm.swap_maxpages pages but the realistic
limit is about half that.  By default the realistic limit is about 4×RAM (on
64-bit architectures), but this can be adjusted via kern.maxswzone (which
defines the #bytes of RAM to allocate to swzone structures - the actual
space allocated is vm.swzone).

As a further piece of arcana, vm.pageout_oom_seq is a count that controls
the number of passes before the pageout daemon gives up and starts killing
processes when it can't free up enough RAM.  "out of swap space" messages
generally mean that this number is too low, rather than there being a
shortage of swap - particularly if your swap device is rather slow.

-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: swap space issues

[Greg 'groggy' Lehey]

On Thursday, 25 June 2020 at 19:31:34 -0700, Donald Wilde wrote:
> On 6/25/20, Greg 'groggy' Lehey  wrote:
>> On Wednesday, 24 June 2020 at 23:27:27 -0700, Kevin Oberman wrote,
>> without trimming:
>>
>>> On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
>>> wrote:
>>
>>> gpart(8) works just fine on MBR drives and partitions/slices and
>>> has a much friendlier user interface. "gpart resize" is the
>>> command you want.
>>
>> Thanks.  I try to offer suggestions that I've tried, and offer an
>> example.  I haven't tried 'gpart resize', but it looks much easier.
>
> 'gpart resize' did work well,

Yes, I saw that from the gpart output you posted.

> although the man page for gpart assumes way too much. I was able to
> successfully work my way through and create ('gpart add') and mount
> not just one but two 32G swap partitions.

Yes, I saw that too.  Not quite what I was suggesting: I suspected
some overflow issue, so the partitions should really have been a
little shy of 32 GB.  And at least for the start you should only mount
one of them.  In the unlikely event that it should threaten to fill
up, you can still mount the other one without rebooting (swapon(1)).

How are things looking now?

Greg
--
Sent from my desktop computer.
Finger g...@freebsd.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA


signature.asc
Description: PGP signature

Re: swap space issues

[Donald Wilde]

On 6/25/20, Greg 'groggy' Lehey  wrote:
> On Wednesday, 24 June 2020 at 23:27:27 -0700, Kevin Oberman wrote,
> without trimming:
>
>> On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
>> wrote:
>
>> gpart(8) works just fine on MBR drives and partitions/slices and has a
>> much
>> friendlier user interface. "gpart resize"  is the command you want.
>
> Thanks.  I try to offer suggestions that I've tried, and offer an
> example.  I haven't tried 'gpart resize', but it looks much easier.
>
'gpart resize' did work well, although the man page for gpart assumes
way too much. I was able to successfully work my way through and
create ('gpart add') and mount not just one but two 32G swap
partitions.
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Greg 'groggy' Lehey]

On Wednesday, 24 June 2020 at 23:27:27 -0700, Kevin Oberman wrote,
without trimming:

> On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
> wrote:
>
>> On Wednesday, 24 June 2020 at 20:34:24 -0700, Donald Wilde wrote:
>>> On 6/24/20, Greg 'groggy' Lehey  wrote:
 On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
> On 6/24/20, Greg 'groggy' Lehey  wrote:
>> On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
>>> All,
...

This would be much more readable if you trimmed unrelated content.
>> That's really puzzling.  It seems that it gave you much more than you
>> asked for.
>>
>> Try this in single user mode: modify the size of the swap partition to
>> 30 GB.  I haven't used MBR partitions for years now, but I believe
>> that 'bsdlabel -e' will do the trick.  Just shorten the length of the
>> b partition.  You may need to 'mount -u /'.  If you do it right
>> (check!), this won't harm any of the other partitions: it'll just
>> leave 26 GB free between the swap partition and the next partition.
>>
> gpart(8) works just fine on MBR drives and partitions/slices and has a much
> friendlier user interface. "gpart resize"  is the command you want.

Thanks.  I try to offer suggestions that I've tried, and offer an
example.  I haven't tried 'gpart resize', but it looks much easier.

Greg
--
Sent from my desktop computer.
Finger g...@freebsd.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA


signature.asc
Description: PGP signature

Re: swap space issues

[Greg 'groggy' Lehey]

On Friday, 26 June 2020 at  1:55:29 +0700, Eugene Grosbein wrote:
> 26.06.2020 1:30, Donald Wilde wrote:
>
>> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>
>> Device  1K-blocks UsedAvail Capacity
>> /dev/ada0s1b 335544320 33554432 0%
>> /dev/ada0s1d 335544320 33554432 0%
>> Total671088640 67108864 0%
>
> "pstat -s" may also be invoked as "swapinfo"

That's no shorter.

> or better "swapinfo -h".

"Better" is in the eye of the beholder.  I find -h extremely
irritating.

Greg
--
Sent from my desktop computer.
Finger g...@freebsd.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA


signature.asc
Description: PGP signature

Re: swap space issues

[Eugene Grosbein]

26.06.2020 1:30, Donald Wilde wrote:

> Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
> 
> Device  1K-blocks UsedAvail Capacity
> /dev/ada0s1b 335544320 33554432 0%
> /dev/ada0s1d 335544320 33554432 0%
> Total671088640 67108864 0%

"pstat -s" may also be invoked as "swapinfo" or better "swapinfo -h".


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/25/20, Paul Mather  wrote:
> On Jun 24, 2020, at 11:34 PM, Donald Wilde  wrote:
>
>
> I think I've missed in this thread where you said which FreeBSD arch you are
> running: is it FreeBSD/amd64 or FreeBSD/i386?  (With an "old" machine, 4 GB
> RAM, and an install still using MBR, it could potentially be FreeBSD/i386.)
>
Sorry, Paul. You're right, I didn't say. amd64. Here's 'uname -a'

FreeBSD Synergy10 12.1-STABLE FreeBSD 12.1-STABLE r362500
LIBERTYSERVER  amd64

> If it is FreeBSD/i386, there is a precedent for it having problems with
> configuring large amounts of swap.  However, it is usually related to having
> relatively little RAM, too (large amounts of swap space means the OS needs
> to use more RAM to keep track of it).
>
That's good to know, Paul. I am once again doing embedded work, which
means I will be coding 32-bit ARM M7's soon enough.

Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):

Device  1K-blocks UsedAvail Capacity
/dev/ada0s1b 335544320 33554432 0%
/dev/ada0s1d 335544320 33554432 0%
Total671088640 67108864 0%





-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Paul Mather]

On Jun 24, 2020, at 11:34 PM, Donald Wilde  wrote:

> Meant that I upgraded from 12.1-RELEASE to 12-STABLE. When I
> configured the -RELEASE install, I manually messed with the MBR disk
> partitions. This is nominally a half-TB HDD which showed up as a total
> of 446 G available (IIRC, gpart should show it's actual size). I did
> auto partitioning, looked at the sizes, and manually set my partitions
> to give me 40G of swap instead of the auto-generated size of 4G.
> 
> This is an old Dell i3 laptop. It's really generic, picked
> specifically as something I could use for Ubuntu or FreeBSD. Dell
> SERVICE TAG is 5K8W162, but it's a generic i3 with 4G of RAM.


I think I've missed in this thread where you said which FreeBSD arch you are 
running: is it FreeBSD/amd64 or FreeBSD/i386?  (With an "old" machine, 4 GB 
RAM, and an install still using MBR, it could potentially be FreeBSD/i386.)

If it is FreeBSD/i386, there is a precedent for it having problems with 
configuring large amounts of swap.  However, it is usually related to having 
relatively little RAM, too (large amounts of swap space means the OS needs to 
use more RAM to keep track of it).

Cheers,

Paul.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Eugene Grosbein]

25.06.2020 18:42, Donald Wilde wrote:

> Thanks, Kevin! My laptop's BIOS is old enough that it balked when I
> tried to boot from a GPT setup of 12.1R. One Of These Days I'll fix
> that but the MBR works and I needed to move on.
> 
> We'll get there! :D

gpart is not for GPT, it is for "GEOM Partitioning".
gpart manages all kind of partitioning in FreeBSD,
including swap partitions.


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/25/20, Kevin Oberman  wrote:
> On Thu, Jun 25, 2020 at 4:42 AM Donald Wilde  wrote:
>
>> On 6/24/20, Kevin Oberman  wrote:
>> > On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
>> > wrote:
>> >
>> >> On Wednesday, 24 June 2020 at 20:34:24 -0700, Donald Wilde wrote:
>> >> > On 6/24/20, Greg 'groggy' Lehey  wrote:
>> >> >> On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
>> >> >>> On 6/24/20, Greg 'groggy' Lehey  wrote:
>> >>  On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
>> >> > All,
>> >> >
>> >> > I recently upgraded my 12-STABLE system to the latest, and now
>> >> > my
>> >> > swap subsystems aren't working. I deliberately set up a 40GB
>> >> > partition for swap, and when I do 'top -t' I am only seeing
>> >> > 7906M
>> >> > total.
>> >> 
>> >>  That looks suspiciously like the difference from 32 GB.  Could it
>> be
>> >>  numeric overflow?  And if so, where?  What does pstat -s say?
>> >> >>>
>> >> >>> Well, hi Greg! LTNT2!
>> >> >>
>> >> >> Indeed.
>> >> >>
>> >> >>> pstat -shm:
>> >> >>>
>> >> >>> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity:
>> 2%
>> >> >>
>> >> >> Now that's really puzzling.  Why does it say 64 G when you said 40
>> >> >> G,
>> >> >> and the error from top tends to confirm it?  How big is the
>> >> >> partition
>> >> >> (gpart output)?
>> >> >
>> >> > Attached 'gpart list' output
>> >>
>> >> FWIW, gpart show would have done the job.  But what I see there is Yet
>> >> Another swap partition size, 66 GB.   So so far we have various parts
>> >> reporting 8 GB, 40 GB, 64 GB and 66 GB.
>> >>
>> >> > Reduced kern.maxswzone to 999. Is it decimal or unlabeled hex?
>> >>
>> >> It'll be decimal, but it refers to the number of swblk structures
>> >> assigned in memory, and after reading the code I'm still not 100% in
>> >> the clear how this relates to the size of swap, if at all.
>> >>
>> >> > 'top' now shows 4597M total swap.
>> >>
>> >> ... and 4.6 GB.  5 different sizes.
>> >>
>> >> You really shouldn't be relying on top for swap info.  It's a third
>> >> party program that demonstrably shows incorrect results (though I
>>
>> I was continuing to reference it because its 'incorrect results' might
>> flag where we need to see things working. When 'top' shows the right
>> results, we've fixed the right thing.
>>
>> >> believe that the maintainer would be very interested to know why and
>> >> to fix it).  But pstat -s (without any further options) should show
>> >> what the kernel thinks.
>>
>> Here's what I see immediately following shutdown -r and boot:
>>
>> Device  1K-blocks UsedAvail Capacity
>> /dev/ada0s1b 671088640 67108864 0%
>>
>> >>
>> >> >>> What else can I share to help diagnose this?
>> >> >>
>> >> >> Background, maybe?  You say that you upgraded your system.  Did you
>> >> >> change the swap size when you did?  What were swap and RAM sizes
>> >> >> before and after?
>> >> >
>> >> > Meant that I upgraded from 12.1-RELEASE to 12-STABLE. When I
>> >> > configured the -RELEASE install, I manually messed with the MBR disk
>> >> > partitions. This is nominally a half-TB HDD which showed up as a
>> >> > total
>> >> > of 446 G available (IIRC, gpart should show it's actual size). I did
>> >> > auto partitioning, looked at the sizes, and manually set my
>> >> > partitions
>> >> > to give me 40G of swap instead of the auto-generated size of 4G.
>> >>
>> >> That's really puzzling.  It seems that it gave you much more than you
>> >> asked for.
>> >>
>> >> Try this in single user mode: modify the size of the swap partition to
>> >> 30 GB.  I haven't used MBR partitions for years now, but I believe
>> >> that 'bsdlabel -e' will do the trick.  Just shorten the length of the
>> >> b partition.  You may need to 'mount -u /'.  If you do it right
>> >> (check!), this won't harm any of the other partitions: it'll just
>> >> leave 26 GB free between the swap partition and the next partition.
>>
>> Thanks again, Greg!
>>
>> >>
>> > gpart(8) works just fine on MBR drives and partitions/slices and has a
>> much
>> > friendlier user interface. "gpart resize"  is the command you want.
>> > --
>> > Kevin Oberman, Part time kid herder and retired Network Engineer
>>
>> Thanks, Kevin! My laptop's BIOS is old enough that it balked when I
>> tried to boot from a GPT setup of 12.1R. One Of These Days I'll fix
>> that but the MBR works and I needed to move on.
>>
>> We'll get there! :D
>>
>
> I think a lot of people miss the point that the fact that gpart was written
> primarily to provide support  GPT partitioning, it also supports MBR and
> bsdlabel is really obsolete. gpart(8) supports 7 different partitioning
> schemes including the old RAW scheme, GPT and MBR. I still have an MBR disk
> on my near decade old laptop and I use gpart on it.

I have been bouncing around between gpart and geom, seeking to
understand them better. The man page is... obtuse... but 

Re: swap space issues

[Kevin Oberman]

On Thu, Jun 25, 2020 at 4:42 AM Donald Wilde  wrote:

> On 6/24/20, Kevin Oberman  wrote:
> > On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
> > wrote:
> >
> >> On Wednesday, 24 June 2020 at 20:34:24 -0700, Donald Wilde wrote:
> >> > On 6/24/20, Greg 'groggy' Lehey  wrote:
> >> >> On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
> >> >>> On 6/24/20, Greg 'groggy' Lehey  wrote:
> >>  On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
> >> > All,
> >> >
> >> > I recently upgraded my 12-STABLE system to the latest, and now my
> >> > swap subsystems aren't working. I deliberately set up a 40GB
> >> > partition for swap, and when I do 'top -t' I am only seeing 7906M
> >> > total.
> >> 
> >>  That looks suspiciously like the difference from 32 GB.  Could it
> be
> >>  numeric overflow?  And if so, where?  What does pstat -s say?
> >> >>>
> >> >>> Well, hi Greg! LTNT2!
> >> >>
> >> >> Indeed.
> >> >>
> >> >>> pstat -shm:
> >> >>>
> >> >>> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity:
> 2%
> >> >>
> >> >> Now that's really puzzling.  Why does it say 64 G when you said 40 G,
> >> >> and the error from top tends to confirm it?  How big is the partition
> >> >> (gpart output)?
> >> >
> >> > Attached 'gpart list' output
> >>
> >> FWIW, gpart show would have done the job.  But what I see there is Yet
> >> Another swap partition size, 66 GB.   So so far we have various parts
> >> reporting 8 GB, 40 GB, 64 GB and 66 GB.
> >>
> >> > Reduced kern.maxswzone to 999. Is it decimal or unlabeled hex?
> >>
> >> It'll be decimal, but it refers to the number of swblk structures
> >> assigned in memory, and after reading the code I'm still not 100% in
> >> the clear how this relates to the size of swap, if at all.
> >>
> >> > 'top' now shows 4597M total swap.
> >>
> >> ... and 4.6 GB.  5 different sizes.
> >>
> >> You really shouldn't be relying on top for swap info.  It's a third
> >> party program that demonstrably shows incorrect results (though I
>
> I was continuing to reference it because its 'incorrect results' might
> flag where we need to see things working. When 'top' shows the right
> results, we've fixed the right thing.
>
> >> believe that the maintainer would be very interested to know why and
> >> to fix it).  But pstat -s (without any further options) should show
> >> what the kernel thinks.
>
> Here's what I see immediately following shutdown -r and boot:
>
> Device  1K-blocks UsedAvail Capacity
> /dev/ada0s1b 671088640 67108864 0%
>
> >>
> >> >>> What else can I share to help diagnose this?
> >> >>
> >> >> Background, maybe?  You say that you upgraded your system.  Did you
> >> >> change the swap size when you did?  What were swap and RAM sizes
> >> >> before and after?
> >> >
> >> > Meant that I upgraded from 12.1-RELEASE to 12-STABLE. When I
> >> > configured the -RELEASE install, I manually messed with the MBR disk
> >> > partitions. This is nominally a half-TB HDD which showed up as a total
> >> > of 446 G available (IIRC, gpart should show it's actual size). I did
> >> > auto partitioning, looked at the sizes, and manually set my partitions
> >> > to give me 40G of swap instead of the auto-generated size of 4G.
> >>
> >> That's really puzzling.  It seems that it gave you much more than you
> >> asked for.
> >>
> >> Try this in single user mode: modify the size of the swap partition to
> >> 30 GB.  I haven't used MBR partitions for years now, but I believe
> >> that 'bsdlabel -e' will do the trick.  Just shorten the length of the
> >> b partition.  You may need to 'mount -u /'.  If you do it right
> >> (check!), this won't harm any of the other partitions: it'll just
> >> leave 26 GB free between the swap partition and the next partition.
>
> Thanks again, Greg!
>
> >>
> > gpart(8) works just fine on MBR drives and partitions/slices and has a
> much
> > friendlier user interface. "gpart resize"  is the command you want.
> > --
> > Kevin Oberman, Part time kid herder and retired Network Engineer
>
> Thanks, Kevin! My laptop's BIOS is old enough that it balked when I
> tried to boot from a GPT setup of 12.1R. One Of These Days I'll fix
> that but the MBR works and I needed to move on.
>
> We'll get there! :D
>

I think a lot of people miss the point that the fact that gpart was written
primarily to provide support  GPT partitioning, it also supports MBR and
bsdlabel is really obsolete. gpart(8) supports 7 different partitioning
schemes including the old RAW scheme, GPT and MBR. I still have an MBR disk
on my near decade old laptop and I use gpart on it.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail 

Re: swap space issues

[Donald Wilde]

On 6/24/20, Kevin Oberman  wrote:
> On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
> wrote:
>
>> On Wednesday, 24 June 2020 at 20:34:24 -0700, Donald Wilde wrote:
>> > On 6/24/20, Greg 'groggy' Lehey  wrote:
>> >> On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
>> >>> On 6/24/20, Greg 'groggy' Lehey  wrote:
>>  On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
>> > All,
>> >
>> > I recently upgraded my 12-STABLE system to the latest, and now my
>> > swap subsystems aren't working. I deliberately set up a 40GB
>> > partition for swap, and when I do 'top -t' I am only seeing 7906M
>> > total.
>> 
>>  That looks suspiciously like the difference from 32 GB.  Could it be
>>  numeric overflow?  And if so, where?  What does pstat -s say?
>> >>>
>> >>> Well, hi Greg! LTNT2!
>> >>
>> >> Indeed.
>> >>
>> >>> pstat -shm:
>> >>>
>> >>> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity: 2%
>> >>
>> >> Now that's really puzzling.  Why does it say 64 G when you said 40 G,
>> >> and the error from top tends to confirm it?  How big is the partition
>> >> (gpart output)?
>> >
>> > Attached 'gpart list' output
>>
>> FWIW, gpart show would have done the job.  But what I see there is Yet
>> Another swap partition size, 66 GB.   So so far we have various parts
>> reporting 8 GB, 40 GB, 64 GB and 66 GB.
>>
>> > Reduced kern.maxswzone to 999. Is it decimal or unlabeled hex?
>>
>> It'll be decimal, but it refers to the number of swblk structures
>> assigned in memory, and after reading the code I'm still not 100% in
>> the clear how this relates to the size of swap, if at all.
>>
>> > 'top' now shows 4597M total swap.
>>
>> ... and 4.6 GB.  5 different sizes.
>>
>> You really shouldn't be relying on top for swap info.  It's a third
>> party program that demonstrably shows incorrect results (though I

I was continuing to reference it because its 'incorrect results' might
flag where we need to see things working. When 'top' shows the right
results, we've fixed the right thing.

>> believe that the maintainer would be very interested to know why and
>> to fix it).  But pstat -s (without any further options) should show
>> what the kernel thinks.

Here's what I see immediately following shutdown -r and boot:

Device  1K-blocks UsedAvail Capacity
/dev/ada0s1b 671088640 67108864 0%

>>
>> >>> What else can I share to help diagnose this?
>> >>
>> >> Background, maybe?  You say that you upgraded your system.  Did you
>> >> change the swap size when you did?  What were swap and RAM sizes
>> >> before and after?
>> >
>> > Meant that I upgraded from 12.1-RELEASE to 12-STABLE. When I
>> > configured the -RELEASE install, I manually messed with the MBR disk
>> > partitions. This is nominally a half-TB HDD which showed up as a total
>> > of 446 G available (IIRC, gpart should show it's actual size). I did
>> > auto partitioning, looked at the sizes, and manually set my partitions
>> > to give me 40G of swap instead of the auto-generated size of 4G.
>>
>> That's really puzzling.  It seems that it gave you much more than you
>> asked for.
>>
>> Try this in single user mode: modify the size of the swap partition to
>> 30 GB.  I haven't used MBR partitions for years now, but I believe
>> that 'bsdlabel -e' will do the trick.  Just shorten the length of the
>> b partition.  You may need to 'mount -u /'.  If you do it right
>> (check!), this won't harm any of the other partitions: it'll just
>> leave 26 GB free between the swap partition and the next partition.

Thanks again, Greg!

>>
> gpart(8) works just fine on MBR drives and partitions/slices and has a much
> friendlier user interface. "gpart resize"  is the command you want.
> --
> Kevin Oberman, Part time kid herder and retired Network Engineer

Thanks, Kevin! My laptop's BIOS is old enough that it balked when I
tried to boot from a GPT setup of 12.1R. One Of These Days I'll fix
that but the MBR works and I needed to move on.

We'll get there! :D
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Kevin Oberman]

On Wed, Jun 24, 2020 at 10:30 PM Greg 'groggy' Lehey 
wrote:

> On Wednesday, 24 June 2020 at 20:34:24 -0700, Donald Wilde wrote:
> > On 6/24/20, Greg 'groggy' Lehey  wrote:
> >> On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
> >>> On 6/24/20, Greg 'groggy' Lehey  wrote:
>  On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
> > All,
> >
> > I recently upgraded my 12-STABLE system to the latest, and now my
> > swap subsystems aren't working. I deliberately set up a 40GB
> > partition for swap, and when I do 'top -t' I am only seeing 7906M
> > total.
> 
>  That looks suspiciously like the difference from 32 GB.  Could it be
>  numeric overflow?  And if so, where?  What does pstat -s say?
> >>>
> >>> Well, hi Greg! LTNT2!
> >>
> >> Indeed.
> >>
> >>> pstat -shm:
> >>>
> >>> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity: 2%
> >>
> >> Now that's really puzzling.  Why does it say 64 G when you said 40 G,
> >> and the error from top tends to confirm it?  How big is the partition
> >> (gpart output)?
> >
> > Attached 'gpart list' output
>
> FWIW, gpart show would have done the job.  But what I see there is Yet
> Another swap partition size, 66 GB.   So so far we have various parts
> reporting 8 GB, 40 GB, 64 GB and 66 GB.
>
> > Reduced kern.maxswzone to 999. Is it decimal or unlabeled hex?
>
> It'll be decimal, but it refers to the number of swblk structures
> assigned in memory, and after reading the code I'm still not 100% in
> the clear how this relates to the size of swap, if at all.
>
> > 'top' now shows 4597M total swap.
>
> ... and 4.6 GB.  5 different sizes.
>
> You really shouldn't be relying on top for swap info.  It's a third
> party program that demonstrably shows incorrect results (though I
> believe that the maintainer would be very interested to know why and
> to fix it).  But pstat -s (without any further options) should show
> what the kernel thinks.
>
> >>> What else can I share to help diagnose this?
> >>
> >> Background, maybe?  You say that you upgraded your system.  Did you
> >> change the swap size when you did?  What were swap and RAM sizes
> >> before and after?
> >
> > Meant that I upgraded from 12.1-RELEASE to 12-STABLE. When I
> > configured the -RELEASE install, I manually messed with the MBR disk
> > partitions. This is nominally a half-TB HDD which showed up as a total
> > of 446 G available (IIRC, gpart should show it's actual size). I did
> > auto partitioning, looked at the sizes, and manually set my partitions
> > to give me 40G of swap instead of the auto-generated size of 4G.
>
> That's really puzzling.  It seems that it gave you much more than you
> asked for.
>
> Try this in single user mode: modify the size of the swap partition to
> 30 GB.  I haven't used MBR partitions for years now, but I believe
> that 'bsdlabel -e' will do the trick.  Just shorten the length of the
> b partition.  You may need to 'mount -u /'.  If you do it right
> (check!), this won't harm any of the other partitions: it'll just
> leave 26 GB free between the swap partition and the next partition.
>
gpart(8) works just fine on MBR drives and partitions/slices and has a much
friendlier user interface. "gpart resize"  is the command you want.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Greg 'groggy' Lehey]

On Wednesday, 24 June 2020 at 20:34:24 -0700, Donald Wilde wrote:
> On 6/24/20, Greg 'groggy' Lehey  wrote:
>> On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
>>> On 6/24/20, Greg 'groggy' Lehey  wrote:
 On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
> All,
>
> I recently upgraded my 12-STABLE system to the latest, and now my
> swap subsystems aren't working. I deliberately set up a 40GB
> partition for swap, and when I do 'top -t' I am only seeing 7906M
> total.

 That looks suspiciously like the difference from 32 GB.  Could it be
 numeric overflow?  And if so, where?  What does pstat -s say?
>>>
>>> Well, hi Greg! LTNT2!
>>
>> Indeed.
>>
>>> pstat -shm:
>>>
>>> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity: 2%
>>
>> Now that's really puzzling.  Why does it say 64 G when you said 40 G,
>> and the error from top tends to confirm it?  How big is the partition
>> (gpart output)?
>
> Attached 'gpart list' output

FWIW, gpart show would have done the job.  But what I see there is Yet
Another swap partition size, 66 GB.   So so far we have various parts
reporting 8 GB, 40 GB, 64 GB and 66 GB.

> Reduced kern.maxswzone to 999. Is it decimal or unlabeled hex?

It'll be decimal, but it refers to the number of swblk structures
assigned in memory, and after reading the code I'm still not 100% in
the clear how this relates to the size of swap, if at all.

> 'top' now shows 4597M total swap.

... and 4.6 GB.  5 different sizes.

You really shouldn't be relying on top for swap info.  It's a third
party program that demonstrably shows incorrect results (though I
believe that the maintainer would be very interested to know why and
to fix it).  But pstat -s (without any further options) should show
what the kernel thinks.

>>> What else can I share to help diagnose this?
>>
>> Background, maybe?  You say that you upgraded your system.  Did you
>> change the swap size when you did?  What were swap and RAM sizes
>> before and after?
>
> Meant that I upgraded from 12.1-RELEASE to 12-STABLE. When I
> configured the -RELEASE install, I manually messed with the MBR disk
> partitions. This is nominally a half-TB HDD which showed up as a total
> of 446 G available (IIRC, gpart should show it's actual size). I did
> auto partitioning, looked at the sizes, and manually set my partitions
> to give me 40G of swap instead of the auto-generated size of 4G.

That's really puzzling.  It seems that it gave you much more than you
asked for.

Try this in single user mode: modify the size of the swap partition to
30 GB.  I haven't used MBR partitions for years now, but I believe
that 'bsdlabel -e' will do the trick.  Just shorten the length of the
b partition.  You may need to 'mount -u /'.  If you do it right
(check!), this won't harm any of the other partitions: it'll just
leave 26 GB free between the swap partition and the next partition.

Greg
--
Sent from my desktop computer.
Finger g...@freebsd.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA


signature.asc
Description: PGP signature

Re: swap space issues

[Donald Wilde]

On 6/24/20, Greg 'groggy' Lehey  wrote:
> On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
>> On 6/24/20, Greg 'groggy' Lehey  wrote:
>>> On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
 All,

 I recently upgraded my 12-STABLE system to the latest, and now my
 swap subsystems aren't working. I deliberately set up a 40GB
 partition for swap, and when I do 'top -t' I am only seeing 7906M
 total.
>>>
>>> That looks suspiciously like the difference from 32 GB.  Could it be
>>> numeric overflow?  And if so, where?  What does pstat -s say?
>>
>> Well, hi Greg! LTNT2!
>
> Indeed.
>
>> pstat -shm:
>>
>> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity: 2%
>
> Now that's really puzzling.  Why does it say 64 G when you said 40 G,
> and the error from top tends to confirm it?  How big is the partition
> (gpart output)?

Attached 'gpart list' output

Reduced kern.maxswzone to 999. Is it decimal or unlabeled hex?

'top' now shows 4597M total swap.

>
 When using synth, it shows swap usage of < 2%, but then it gives the
 'out of swap' complaint and tosses all my users out into neverland.
>>>
>>> Unless this also causes kernel messages to that effect, this, too,
>>> could be an issue with synth.
>>
>> I believe the messages about 'out of swap' were from the kernel as
>> it died, not synth. As I say, it killed all my active users, both
>> root and non-root.

Okay, I've now got data on that. It crashed again. :-\

First, the kernel killed init-spawned PIDs up into the 12000s: 'out of
swap space'.

Next, I get the following 5 times, boldface (this is console, not a GUI):

swblk zone ok
swap blk zone exhausted, increase kern.maxswzone

Back to normal font, the kernel killed four cron-job PIDs, 'out of swap space'

At this point, if I hit a , I get a login prompt.

>
> Hmm.
>
>> Why does _top_ show the 7906M number? I suspect that it's deeper
>> than just synth.
>
> If top counts swap in a form that overflows at 32 GB, 40 GB could show
> up as 8 GB.
>
>> What else can I share to help diagnose this?
>
> Background, maybe?  You say that you upgraded your system.  Did you
> change the swap size when you did?  What were swap and RAM sizes
> before and after?

Meant that I upgraded from 12.1-RELEASE to 12-STABLE. When I
configured the -RELEASE install, I manually messed with the MBR disk
partitions. This is nominally a half-TB HDD which showed up as a total
of 446 G available (IIRC, gpart should show it's actual size). I did
auto partitioning, looked at the sizes, and manually set my partitions
to give me 40G of swap instead of the auto-generated size of 4G.

This is an old Dell i3 laptop. It's really generic, picked
specifically as something I could use for Ubuntu or FreeBSD. Dell
SERVICE TAG is 5K8W162, but it's a generic i3 with 4G of RAM.

I'm developing a multi-server application for my cousin, so I'm using
this as a mule to work my way through some of the issues involved in
that project, from FreeBSD itself through NGINX and Lua, etc.
>
> FWIW, I've had a very large number of issues with -STABLE in recent
> months, though not this one.  I should send a separate message.
>
I guess I've been fortunate, then. Most of my issues have been with
re-learning FreeBSD. :D
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 976773167
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: ada0s1
   Mediasize: 500107829248 (466G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e3
   efimedia: HD(1,MBR,0x90909090,0x40,0x3a385ff0)
   attrib: active
   rawtype: 165
   length: 500107829248
   offset: 32768
   type: freebsd
   index: 1
   end: 976773167
   start: 64
Consumers:
1. Name: ada0
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: ada0s1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 976773103
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s1a
   Mediasize: 429496729600 (400G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawtype: 7
   length: 429496729600
   offset: 0
   type: freebsd-ufs
   index: 1
   end: 838860799
   start: 0
2. Name: ada0s1b
   Mediasize: 70611099648 (66G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e0
   rawtype: 1
   length: 70611099648
   offset: 429496729600
   type: freebsd-swap
   index: 2
   end: 976773103
   start: 838860800
Consumers:
1. Name: ada0s1
   Mediasize: 500107829248 (466G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e3

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To 

Re: swap space issues

[Greg 'groggy' Lehey]

On Wednesday, 24 June 2020 at 18:51:04 -0700, Donald Wilde wrote:
> On 6/24/20, Greg 'groggy' Lehey  wrote:
>> On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
>>> All,
>>>
>>> I recently upgraded my 12-STABLE system to the latest, and now my
>>> swap subsystems aren't working. I deliberately set up a 40GB
>>> partition for swap, and when I do 'top -t' I am only seeing 7906M
>>> total.
>>
>> That looks suspiciously like the difference from 32 GB.  Could it be
>> numeric overflow?  And if so, where?  What does pstat -s say?
>
> Well, hi Greg! LTNT2!

Indeed.

> pstat -shm:
>
> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity: 2%

Now that's really puzzling.  Why does it say 64 G when you said 40 G,
and the error from top tends to confirm it?  How big is the partition
(gpart output)?

>>> When using synth, it shows swap usage of < 2%, but then it gives the
>>> 'out of swap' complaint and tosses all my users out into neverland.
>>
>> Unless this also causes kernel messages to that effect, this, too,
>> could be an issue with synth.
>
> I believe the messages about 'out of swap' were from the kernel as
> it died, not synth. As I say, it killed all my active users, both
> root and non-root.

Hmm.

> Why does _top_ show the 7906M number? I suspect that it's deeper
> than just synth.

If top counts swap in a form that overflows at 32 GB, 40 GB could show
up as 8 GB.

> What else can I share to help diagnose this?

Background, maybe?  You say that you upgraded your system.  Did you
change the swap size when you did?  What were swap and RAM sizes
before and after?

FWIW, I've had a very large number of issues with -STABLE in recent
months, though not this one.  I should send a separate message.

Greg
--
Sent from my desktop computer.
Finger g...@freebsd.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA


signature.asc
Description: PGP signature

Re: swap space issues

[Donald Wilde]

On 6/24/20, Donald Wilde  wrote:
> On 6/24/20, Greg 'groggy' Lehey  wrote:
>> On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
>>> All,
>>>
>>> I recently upgraded my 12-STABLE system to the latest, and now my
>>> swap subsystems aren't working. I deliberately set up a 40GB
>>> partition for swap, and when I do 'top -t' I am only seeing 7906M
>>> total.
>>
>> That looks suspiciously like the difference from 32 GB.  Could it be
>> numeric overflow?  And if so, where?  What does pstat -s say?
>>
>
> Well, hi Greg! LTNT2!
>
> pstat -shm:
>
> /dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity: 2%
>
> This is while synth is still running on openblas and openjdk8.
>
>>> When using synth, it shows swap usage of < 2%, but then it gives the
>>> 'out of swap' complaint and tosses all my users out into neverland.
>>
>> Unless this also causes kernel messages to that effect, this, too,
>> could be an issue with synth.
>
> I believe the messages about 'out of swap' were from the kernel as it
> died, not synth. As I say, it killed all my active users, both root
> and non-root.
>
> Why does _top_ show the 7906M number? I suspect that it's deeper than
> just synth.
>
> What else can I share to help diagnose this?

Let me try this: I can reduce kern.maxswzone to a number lower than 32G.

That won't _solve_ the underlying problem, but we can see if it
impacts both top and synth.

I just did portsnap fetch update and it came up with 300 patches, so
that should give us a good run!

[snip]

-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/24/20, Greg 'groggy' Lehey  wrote:
> On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
>> All,
>>
>> I recently upgraded my 12-STABLE system to the latest, and now my
>> swap subsystems aren't working. I deliberately set up a 40GB
>> partition for swap, and when I do 'top -t' I am only seeing 7906M
>> total.
>
> That looks suspiciously like the difference from 32 GB.  Could it be
> numeric overflow?  And if so, where?  What does pstat -s say?
>

Well, hi Greg! LTNT2!

pstat -shm:

/dev/ada0s1b  65536 (1M blocks), Used: 1.5G, Avail: 63G, Capacity: 2%

This is while synth is still running on openblas and openjdk8.

>> When using synth, it shows swap usage of < 2%, but then it gives the
>> 'out of swap' complaint and tosses all my users out into neverland.
>
> Unless this also causes kernel messages to that effect, this, too,
> could be an issue with synth.

I believe the messages about 'out of swap' were from the kernel as it
died, not synth. As I say, it killed all my active users, both root
and non-root.

Why does _top_ show the 7906M number? I suspect that it's deeper than
just synth.

What else can I share to help diagnose this?

>
> Greg
> --
> Sent from my desktop computer.
[snip]
-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Donald Wilde]

On 6/24/20, Donald Wilde  wrote:
> All,
>
> I recently upgraded my 12-STABLE system to the latest, and now my swap
> subsystems aren't working. I deliberately set up a 40GB partition for
> swap, and when I do 'top -t' I am only seeing 7906M total.
>
> When using synth, it shows swap usage of < 2%, but then it gives the
> 'out of swap' complaint and tosses all my users out into neverland.
>
> My /boot/loader.conf has:
>
>kern.maxproc=75000
>kern.maxswzone=1720
>
>... but those are no longer effective.
>
> What has changed, and what do I need to do to rectify this?

I thought I had it running well... 3 tasks and max 8 builders / task.
It was sitting at 0.0 - 1% swap. Perking right along. Looked away to
another bit of work and it has blyecch'd again.

Rebooted, starting it again. It does seem to start over from where it
crashed, which is good.

I'm not in any way saying that this is a bad tool, just trying to
figure out why it crashes even with all the resources I've given it.

Building math/openblas and devel/llvm90 simultaneously, so it's
obvious what put it over the top!

Sigh... barfed again, but at least this time it marked those as failed
and started on new ports. I guess if I keep repeating the process
it'll get through them all. :D

-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: swap space issues

[Greg 'groggy' Lehey]

On Wednesday, 24 June 2020 at  9:36:23 -0700, Donald Wilde wrote:
> All,
>
> I recently upgraded my 12-STABLE system to the latest, and now my
> swap subsystems aren't working. I deliberately set up a 40GB
> partition for swap, and when I do 'top -t' I am only seeing 7906M
> total.

That looks suspiciously like the difference from 32 GB.  Could it be
numeric overflow?  And if so, where?  What does pstat -s say?

> When using synth, it shows swap usage of < 2%, but then it gives the
> 'out of swap' complaint and tosses all my users out into neverland.

Unless this also causes kernel messages to that effect, this, too,
could be an issue with synth.

Greg
--
Sent from my desktop computer.
Finger g...@freebsd.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA


signature.asc
Description: PGP signature

swap space issues

[Donald Wilde]

All,

I recently upgraded my 12-STABLE system to the latest, and now my swap
subsystems aren't working. I deliberately set up a 40GB partition for
swap, and when I do 'top -t' I am only seeing 7906M total.

When using synth, it shows swap usage of < 2%, but then it gives the
'out of swap' complaint and tosses all my users out into neverland.

My /boot/loader.conf has:

   kern.maxproc=75000
   kern.maxswzone=1720

   ... but those are no longer effective.

What has changed, and what do I need to do to rectify this?

-- 
Don Wilde

* What is the Internet of Things but a system *
* of systems including humans? *

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"