The good news is that the problem is not subtle and I have been able to reproduce the lockup. And several times, if just barely provoked, the system came back. At least once, it didn't come back.
I created a netbsd-current domU (pvhvm) with 6G RAM xbd0: 32G ffs2 root xbd1: 8G swap xbd2: 32G gpt with one big zfs partiition tank11: pool with just dk0 from xbd2 Not sure it matters, but the backing disks for the xbdN are zvol in zfs on dom0, on a not particularly new Sandisk 1T SATA SSD. I wrote a script: create 100 dirs with 100 files each sync sleep 10 remove the files sync Long ago I wrote a program "touchmem" to allocate a specific amount of memory, writing into each page to force allocation. I found that the removal process was slow, and if I ran touchmem 6000 (to allocate 6000K) I would get on the console (this is an example where it came back). [ 2247.3254720] arc_reclaim_thread: negative free_memory -15888384 [ 2247.5554953] arc_reclaim_thread: negative free_memory -15908864 [ 2247.7955123] arc_reclaim_thread: negative free_memory -15814656 [ 2248.0355384] arc_reclaim_thread: negative free_memory -15839232 [ 2248.5255792] arc_reclaim_thread: negative free_memory -15847424 [ 2249.2455995] arc_reclaim_thread: negative free_memory -15814656 [ 2250.2756439] arc_reclaim_thread: negative free_memory -15835136 [ 2251.2756950] arc_reclaim_thread: negative free_memory -15847424 [ 2252.2757175] arc_reclaim_thread: negative free_memory -15847424 [ 2252.3257269] arc_reclaim_thread: negative free_memory -15867904 [ 2252.3357294] arc_reclaim_thread: negative free_memory -15876096 [ 2252.3457327] arc_reclaim_thread: negative free_memory -15958016 [ 2252.3657417] arc_reclaim_thread: negative free_memory -15867904 [ 2252.9757657] arc_reclaim_thread: negative free_memory -15835136 [ 2253.9858155] arc_reclaim_thread: negative free_memory -15851520 [ 2254.9958551] arc_reclaim_thread: negative free_memory -15847424 [ 2255.9958915] arc_reclaim_thread: negative free_memory -15839232 [ 2256.9959338] arc_reclaim_thread: negative free_memory -15831040 [ 2257.3259677] arc_reclaim_thread: negative free_memory -15859712 [ 2257.3459560] arc_reclaim_thread: negative free_memory -15831040 [ 2257.3759883] arc_reclaim_thread: negative free_memory -15855616 [ 2257.3859963] arc_reclaim_thread: negative free_memory -15831040 [ 2257.4260042] arc_reclaim_thread: negative free_memory -15839232 [ 2258.4360174] arc_reclaim_thread: negative free_memory -15831040 [ 2259.4360530] arc_reclaim_thread: negative free_memory -15851520 [ 2260.4360959] arc_reclaim_thread: negative free_memory -15831040 [ 2261.4361236] arc_reclaim_thread: negative free_memory -15835136 [ 2262.4361659] arc_reclaim_thread: negative free_memory -15843328 [ 2263.4362053] arc_reclaim_thread: negative free_memory -15831040 [ 2264.4362512] arc_reclaim_thread: negative free_memory -15831040 [ 2265.4362893] arc_reclaim_thread: negative free_memory -15839232 [ 2266.4363080] arc_reclaim_thread: negative free_memory -15855616 [ 2267.4363426] arc_reclaim_thread: negative free_memory -15831040 [ 2268.4463773] arc_reclaim_thread: negative free_memory -15843328 [ 2269.4464351] arc_reclaim_thread: negative free_memory -15847424 [ 2270.4464547] arc_reclaim_thread: negative free_memory -15831040 [ 2271.4465084] arc_reclaim_thread: negative free_memory -15843328 [ 2272.4465497] arc_reclaim_thread: negative free_memory -15839232 [ 2273.4565874] arc_reclaim_thread: negative free_memory -15831040 [ 2274.4566118] arc_reclaim_thread: negative free_memory -15839232 [ 2275.4566479] arc_reclaim_thread: negative free_memory -15843328 [ 2276.4566869] arc_reclaim_thread: negative free_memory -15835136 [ 2277.4567207] arc_reclaim_thread: negative free_memory -15839232 [ 2278.4567630] arc_reclaim_thread: negative free_memory -15831040 [ 2279.4568163] arc_reclaim_thread: negative free_memory -15839232 [ 2280.4568310] arc_reclaim_thread: negative free_memory -15835136 [ 2281.4668732] arc_reclaim_thread: negative free_memory -15843328 [ 2282.4769259] arc_reclaim_thread: negative free_memory -15851520 [ 2283.5369614] arc_reclaim_thread: negative free_memory -15831040 [ 2284.5470001] arc_reclaim_thread: negative free_memory -15835136 [ 2285.5570314] arc_reclaim_thread: negative free_memory -15835136 [ 2286.5671034] arc_reclaim_thread: negative free_memory -7159808 [ 2287.5671318] arc_reclaim_thread: negative free_memory -851968 So the needed combination seems to be - lots of removals (may be just metadata changes) - memory pressure I'll put my test programs up someplace and look at arc_reclaim_thread. I wonder if others who have problems also see this kernel message.
