Re: FYI: example "panic: ARM64TODO: reclaim_pv_chunk" on a Pine64+ 2GB with head -r317015 [mkimg okay; "cp -p" generates the bad context]

2017-04-30 Thread Mark Millard 
On 2017-Apr-27, at 10:26 PM, Mark Millard  wrote:

> [As the text does not really follow from the earlier text
> I'd sent directly I'm top posting a hypothesis about where
> so much active memory came from to be so low in available
> memory to get an reclaim_pv_chunk attempt.]
> 
> My hypothesis is that the "mdconfig -d"s from vm_copy_base
> (from /usr/src/release/tools/vmimage.subr ) did not actually
> release the memory resources involved (from vnode backed
> mdconfig use):

I watched with "vmstat 1" and "mdconfig -d" did release memory
(including RAM) each time.

> . . .
> From the prior top report for the failure,
> partially repeated here:
> 
> . . .
> Mem: 1618M Active, 17M Inact, 315M Wired, 204M Buf, 15M Free
> Swap: 6144M Total, 34M Used, 6110M Free, 348K Out
> 
>  PID USERNAMETHR PRI NICE   SIZERES   SWAP STATE   C   TIME CPU 
> COMMAND
> 48988 root  4  310   651M 27048K 0K RUN 0   0:03  87.60% 
> xz -T 0 /usr/obj/DESTDIRs/vmimage-aarch64/vmimages/FreeBSD-12.0-CURRENT-
> . . .
> 
> The combination 1618M Mem:Active but 34M Swap:Used
> and 651M xz:SIZE but 27M xz:RES and 0K xz:SWAP just
> seems very odd, like it should not happen. The 17M
> Mem:Inact is odd for the context as well. (Mem:Wired,
> Mem:Buf, and Mem:Free look normal.)
> 
> An alternate hypothesis would be the memory "leak"
> is from mkimg not having it memory-use cleaned up.
> This happens after vm_copy_base but before the cp/xz
> sequence and is what produces vm.raw.

mkimg also release its memory (including RAM)
each time.

But the later "cp -p" of the large vm.raw that
I was producing ended up without leaving the
free memory that is expected after it finished.
I worked around the issue with (just a personal
workaround that helps in other respects as well):

Index: Makefile.vm
===
--- Makefile.vm (revision 317015)
+++ Makefile.vm (working copy)
@@ -119,15 +119,20 @@
 vm-install:
 .if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES)
mkdir -p ${DESTDIR}/vmimages
-. for FORMAT in ${VMFORMATS}
-   cp -p ${VMBASE}.${FORMAT} \
-   ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}
-. endfor
 . if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES)
 .  for FORMAT in ${VMFORMATS}
-   # Don't keep the originals.  There is a copy in ${.OBJDIR} if needed.
-   ${XZ_CMD} ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}
+   # Tradeoff "cp -p" property for less memory, I/O, and time use.
+   # Also: -3 got > 30 MiByte/sec effective (source file) for 32 GiByte, 
mostly-zero image
+   # on a Pine64+ 2GB and used about 600 MiBytes of "active virtual 
memory". The about 16
+   # minutes was vastly faster than the "cp -p" --and the sha512 and 
sha256 are vastly
+   # faster on the compressed file as well.
+   ${XZ_CMD} -T 0 -3 --stdout --memlimit-compress=50% -v 
${VMBASE}.${FORMAT} > ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}.xz
 .  endfor
+. else
+.  for FORMAT in ${VMFORMATS}
+   cp -p ${VMBASE}.${FORMAT} \
+   ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}
+.  endfor
 . endif
cd ${DESTDIR}/vmimages && sha512 ${OSRELEASE}* > \
${DESTDIR}/vmimages/CHECKSUM.SHA512

and using WITH_COMPRESSED_VMIMAGES to avoid a "cp -p"
based copy of the large file.


Prior reports from capturing text:

On 2017-Apr-27, at 7:31 PM, Mark Millard  wrote:

> [Another example panic. Again no dump. But I have what
> a top -PCwaopid froze at this time.]
> 
> On 2017-Apr-27, at 4:22 PM, Mark Millard  wrote:
> 
>> Unfortunately for this FYI the attempt to produce a dump
>> failed and so all the information that I have is what I
>> first captured from the console output: a backtrace.
>> 
>> The context was head -r317015 on a Pine64+ 2GB. At the
>> time I was experimenting with trying to build a vm.raw
>> from my own build of FreeBSD. The (root) file system
>> is on a USB SSD off of a powered USB hub.
>> 
>> panic: ARM64TODO: reclaim_pv_chunk
>> cpuid = 1
>> time = 1493332968
>> KDB: stack backtrace:
>> db_trace_self() at db_trace_self_wrapper+0x28
>>   pc = 0x00605cc0  lr = 0x000869cc
>>   sp = 0x83ba4f00  fp = 0x83ba5110
>> 
>> db_trace_self_wrapper() at vpanic+0x164
>>   pc = 0x000869cc  lr = 0x0031d464
>>   sp = 0x83ba5120  fp = 0x83ba5190
>> 
>> vpanic() at panic+0x4c
>>   pc = 0x0031d464  lr = 0x0031d2fc
>>   sp = 0x83ba51a0  fp = 0x83ba5220
>> 
>> panic() at reclaim_pv_chunk+0x10
>>   pc = 0x0031d2fc  lr = 0x0061a234
>>   sp = 0x83ba5230  fp = 0x83ba5230
>> 
>> reclaim_pv_chunk() at get_pv_entry+0x240
>>   pc = 0x0061a234  lr = 0x00616184
>>   sp = 0x83ba5240  fp = 0x83ba5260
>> 
>> get_pv_entry() at pmap_enter+0x694
>>   pc = 0x00616184  lr = 0x006156a0
>>   sp =

Re: FYI: FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw under qemu-system-aarch64 on odroid-c2 under UbuntuMate : No valid device tree blob found!

2017-04-30 Thread Mark Millard 
On 2017-Apr-30, at 9:40 AM, Andrew Turner  wrote:

>> On 30 Apr 2017, at 12:02, Mark Millard  wrote:
>> 
>> On 2017-Apr-30, at 1:57 AM, Andrew Turner  wrote:
>> 
 On 30 Apr 2017, at 04:29, Mark Millard  wrote:
 ...
 acpi0: 
 acpi0: Power Button (fixed)
 acpi0: Sleep Button (fixed)
>>> 
>>> ACPI is not fully supported on arm64.
>> 
>> Good to know. Thanks.
>> 
>> But the messages:
>> 
>> No valid device tree blob found!
>> WARNING! Trying to fire up the kernel, but no device tree blob found!
>> 
>> were well before the acpi0 messages so I'd expect
>> that the lack of a "device tree blob" is a separate,
>> earlier issue, likely to do with the content of:
>> 
>> FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw
> 
> No, the device tree blob comes from UEFI. It seems the current UEFI only 
> provides the ACPI tables, and not a DTB.

So you are expecting that the older QEMU_EFI.fd I had
used before provided some sort of fairly generic dtb
(relative to qemu, fairly independent of the host
that was running qemu). Interesting. Thanks again.

It turns out that if I put the odroid-c2's dtb in where
I can load it from the loader prompt (e.g., in /boot/dtb/ )
I get a little farther, although with notices for:

Failed to start CPU 1 (1)
Failed to start CPU 2 (2)
Failed to start CPU 3 (3)
pmu0: could not allocate resources
device_attach: pmu0 attach returned 6
gpioled0:  failed to map pin
usb_needs_explore_all: no devclass

The details of the sequence are:

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel] in 9 seconds... 

Type '?' for a list of commands, 'help' for more detailed help.
OK load -t dtb /boot/dtb/meson64_odroidc2.dtb
/boot/dtb/meson64_odroidc2.dtb size=0x746c
OK boot
?[37m?[44mBooting...?[m
Using DTB from loaded file '/boot/dtb/meson64_odroidc2.dtb'.
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT  r317015M arm64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 
4.0.0)
VT: init without driver.
Starting CPU 1 (1)
Failed to start CPU 1 (1)
Starting CPU 2 (2)
Failed to start CPU 2 (2)
Starting CPU 3 (3)
Failed to start CPU 3 (3)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
arc4random: no preloaded entropy cache
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: 
psci0:  on ofwbus0
gic0:  mem 
0xc4301000-0xc4301fff,0xc4302000-0xc4303fff,0xc4304000-0xc4305fff,0xc4306000-0xc4307fff
 irq 18 on ofwbus0
gic0: pn 0x0, arch 0x0, rev 0x0, implementer 0x0 irqs 32
generic_timer0:  irq 0,1,2,3 on ofwbus0
Timecounter "ARM MPCore Timecounter" frequency 2400 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 2400 Hz quality 1000
cpulist0:  on ofwbus0
cpu0:  on cpulist0
cpu1:  on cpulist0
cpu2:  on cpulist0
cpu3:  on cpulist0
pmu0:  irq 4,5,6,7 on ofwbus0
pmu0: could not allocate resources
device_attach: pmu0 attach returned 6
gpioled0:  on ofwbus0
gpioled0:  failed to map pin
cryptosoft0: 
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass

As of that it hangs.

Using something like pine64_plus.dtb did not get
far at all compared to the above.

It is possible that for "-enable-kvm -cpu host" that
the dtb needs to match the host machine's dtb.

Still, it is not like what I thought I remembered from
back in 2016-September: I did no dtb manipulations
back then since I was unaware of the issue.

The above is console output is for. . .

qemu-system-aarch64 -m 1024M -enable-kvm -cpu host -M virt \
-bios QEMU_EFI.fd -nographic \
-drive 
format=raw,if=none,file=FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw,id=hd0
 \
-device virtio-blk-device,drive=hd0 \
-device virtio-net-device,netdev=net0 \
-netdev user,id=net0 \
-smp cpus=4

based on:

http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/1917/QEMU-AARCH64/RELEASE_CLANG35/QEMU_EFI.fd

but this time based on my build of head -r317015 .

I first got to the point of replicating what I'd
reported for:

FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw

then I added the 2 .dtb files to try loading. The
Odroid-C2 one worked better on the Odroid-C2.

It seems that qemu does not have a general dtb
emulation capability. (No surprise there?) But
I did not find anything saying what was required.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw under qemu-system-aarch64 on odroid-c2 under UbuntuMate : No valid device tree blob found!

2017-04-30 Thread Andrew Turner 

> On 30 Apr 2017, at 12:02, Mark Millard  wrote:
> 
> On 2017-Apr-30, at 1:57 AM, Andrew Turner  wrote:
> 
>>> On 30 Apr 2017, at 04:29, Mark Millard  wrote:
>>> ...
>>> acpi0: 
>>> acpi0: Power Button (fixed)
>>> acpi0: Sleep Button (fixed)
>> 
>> ACPI is not fully supported on arm64.
> 
> Good to know. Thanks.
> 
> But the messages:
> 
> No valid device tree blob found!
> WARNING! Trying to fire up the kernel, but no device tree blob found!
> 
> were well before the acpi0 messages so I'd expect
> that the lack of a "device tree blob" is a separate,
> earlier issue, likely to do with the content of:
> 
> FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw

No, the device tree blob comes from UEFI. It seems the current UEFI only 
provides the ACPI tables, and not a DTB.

Andrew

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw under qemu-system-aarch64 on odroid-c2 under UbuntuMate : No valid device tree blob found!

2017-04-30 Thread Mark Millard 
On 2017-Apr-30, at 1:57 AM, Andrew Turner  wrote:

>> On 30 Apr 2017, at 04:29, Mark Millard  wrote:
>> ...
>> acpi0: 
>> acpi0: Power Button (fixed)
>> acpi0: Sleep Button (fixed)
> 
> ACPI is not fully supported on arm64.

Good to know. Thanks.

But the messages:

No valid device tree blob found!
WARNING! Trying to fire up the kernel, but no device tree blob found!

were well before the acpi0 messages so I'd expect
that the lack of a "device tree blob" is a separate,
earlier issue, likely to do with the content of:

FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw

My old 2016-Sep. notes showed no such notifications
for the 11-RELEASE attempt I made back then.


(After the acpi0 messages:) I guessed that the "29"s in:

generic_timer0:  irq 29,30,27 on acpi0
panic: Attempt to copy invalid resource id: 29

were the same "29"s: i.e., that the "resource id" was the
generic_time0: 29 irq number.

But being after the acpi0 messages and after the notice
of a lack of a "device tree blob" may be things are
already messed up by that point --or may be my guess
about the "29"s is just wrong and "resource id" 29
is something else, possibly related to the acpi0
notices.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: FreeBSD-12.0-CURRENT-arm64-aarch64-20170420-r317181.raw under qemu-system-aarch64 on odroid-c2 under UbuntuMate : No valid device tree blob found!

2017-04-30 Thread Andrew Turner 

> On 30 Apr 2017, at 04:29, Mark Millard  wrote:
> ...
> acpi0: 
> acpi0: Power Button (fixed)
> acpi0: Sleep Button (fixed)

ACPI is not fully supported on arm64.

Andrew

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"