hi folks, as taylor requested after the previous mail about this, I've made a copy of the freebsd files that I started with and put them in the netbsd source tree layout so that they can easily be diffed against the merged tree. I've put several files in http://ftp.netbsd.org/pub/NetBSD/misc/chs/cddl/20171205/ :
f-n-remap.pl the perl script to rearrange the cddl files from a freebsd source tree into the netbsd source layout (mostly directory moves, a few file renames) fbsd-r315983.tar.gz the freebsd sources that I started with fbsd-r315983-remapped.tar.gz those freebsd sources as rearranged by the script diff.cddl-fbsd-nbsd diff of fbsd-r315983-remapped.tar.gz vs. the merged tree diff.cddl.20171205 cvs diff of netbsd -current vs. the merged tree this version also includes some cleanup of the existing files, mainly removing about 30 mostly useless glue header files (which I found in the process of getting the remap script working). I haven't conditionalized DTRACE_OPTS yet, but otherwise it should be very close to what I'll commit. please take a look and let me know if there are any other changes that are desired before I commit it. -Chuck On Fri, Oct 13, 2017 at 09:56:01AM -0700, Chuck Silvers wrote: > hi folks, > > I've been working on updating netbsd's copy of the dtrace and zfs code to > rebase > from the existing ancient opensolaris version to a recent freebsd version. > most of the freebsd changes are pretty close to what netbsd needs, > so that seems like a more useful upstream for us. I have things working > well enough now that I want to share the code in preparation for committing. > > this update improves upon our existing dtrace/zfs code in several ways: > - picks up all the upstream zfs fixes and enhancements from the last decade > - zfs now supports mmap on netbsd, so you can run executables stored in zfs > - dtrace fbt probes can now be used in kernel modules (such as zfs) > > here's the diff, which is based on netbsd HEAD as of > "Tue Oct 10 20:14:48 UTC 2017": > http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.cddl.20171012 > > note that the patch needs to be applied with "patch -E" since it > adds and removes various files. > > here's a summary of what's in the diff: > > - freebsd's dtrace/zfs code as of r315983 (2017-03-26), adapted for netbsd. > many netbsd changes in our current version to fix warnings have been > removed > or redone to be reduce the difference from the new upstream. > - a few updates to our copy of freebsd's libproc. > I plan to do a wholesale resync of libproc to current freebsd > rather than just picking up the few bits in this diff. > - build system support for using -fno-omit-frame-pointer everywhere > and disabling other compiler optimizations that confuse dtrace. > I'm ok with leaving these changes disabled by default, > or perhaps making them conditional on MKDTRACE. > - sample kernel config changes for a couple evbarm configs (the ones I > tested). > - module/ksyms enhancements for dtrace integration. > - genfs API enhancements to support zfs. > - an option to have mutexes not become no-ops during a panic. > - uvm_aobj API change to support 64-bit aobj sizes (eg. for tmpfs). > > the plan is that items listed separately above will be committed separately > but in quick succession, to allow one kernel version increment for all of > the ones which require that. > > > here's a list of known issues with the patch: > > - unloading the zfs module fails even with no zpools imported > if you've done much with zfs since it was loaded. > there's some refcounting problem that I haven't tracked down yet. > > - the module refcounting for active fbt probes is bogus. > currently module refcounting is protected by kernconfig_lock(), > but taking that lock down in the bowels of dtrace seems likely > to create deadlocks. I plan to do something fancier but haven't > gotten to it yet. > > - the dtrace uregs[] stuff is probably still wrong. > > - the CTF typeid overflow problem is still there (more on this below). > > - a bunch of features are still not supported: > - the ".zfs" virtual directory, eg. ".zfs/snapshot/foo@bar" > - zvols > - ZFS ACLs (aka. NFSv4 ACLs) > - NFS exporting a ZFS file system > - setting dtrace probes in application code > - using ZFS as the root fs > - new crypto hashes SHA512_256, skein, and edonr > (the last one is not in freebsd yet either) > - zio delay injection (used for testing zfs) > - dtrace support for platforms other than x86 and arm > > > I don't intend to fix any of these issues before initial commit of > these changes, since I don't think any of them are regressions from > what we have currently. > > there a few other bits still in the diff that are just for debugging or > otherwise need to be cleaned up, but not very many. I'll clean them up > before committing. > > > about the CTF typeid overflow issue I mentioned above: > > CTF is the Compact Type Format data that dtrace uses to know about > function argument types and return values. CTF encodes a type ID > as a 15-bit value, but unfortunately a netbsd GENERIC kernel has > more than 15 bits worth of types, so the values overflow the field. > we don't really want to change this format unilaterally since it is > currently the same on all dtrace-supporting operating systems, so we > need to work around the problem for now by removing various subsystems > from the kernel config used with dtrace, so that the number of types > is within what CTF supports. > > to tell if your kernel will work with dtrace, run > ctdump /path/to/kernel | grep "total number of types" > > if the number of types is less than 32768 then you're fine, > otherwise you'll need to change your kernel config to remove > some drivers or file systems that you won't use. > > this error is another symptom of the typeid overflow problem: > dtrace: failed to compile script sleep.d: "/usr/lib/dtrace/psinfo.d", line > 46: failed to copy type of 'pr_arglen': Type information is in parent and > unavailable > > note that dtrace scripts that do not reference any argument or return types > will work even if the kernel has too many types for CTF (but you'll need to > remove the psinfo.d file mentioned above to remove that reference). > > I hope to integrate a solution for this CTF overflow issue before > the first netbsd release with this new dtrace code in it (netbsd 9.0), > but it's not clear what that solution will end up being yet. > > > anyone interested in zfs or dtrace on netbsd please give this a try > and let me know about any problems that you find. > any comments on the diff are also welcome. > > -Chuck