Hi, Adding the systemtap list which has some smart people on it who can probably help out.
Disclaimer: I have played a bit and hacked a bit on Systemtap and as Andrew said I did make dtrace probes in hotspot work nicely together with Systemtap a couple of years ago: http://gnu.wildebeest.org/blog/mjw/2009/08/06/i-love-it-when-a-plan-comes-together-systemtap-meets-java/ But I never have used macosx and the few times I tried to use Solaris/OpenIndiana dtrace it crashed my system. So I don't really know much about dtrace itself. When I added the dtrace SDT probe support on GNU/Linux I did ask for the hotspot/dtrace testcases to make sure everything was working well. But these testcases are apparently secret making it hard to make any compatibility claims :{ On Fri, Sep 06, 2013 at 11:22:42PM -0400, Andrew Hughes wrote: > > I'm working with a GSoC student adding dtrace probe support to JRuby, > > and ran into the issue that user-defined dtrace probes only work on > > Solaris. So I started poking around hotspot source and found that both > > the linux and bsd JSDT sources are empty and return false for > > isSupported. Only solaris has JSDT support. > > > Linux doesn't use DTrace. It uses SystemTap. > > CCing Mark Wielaard who added this support: > > http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/rev/c707a5af0d71 "added support" is a bit much for that simple hack. The support was always there, all I did was tweak the Makefile a bit to get it compiled in (yes, that simple patch took months to get accepted, but that wasn't because the patch was deep or difficult). On GNU/Linux sys/sdt.h probes are available to systemtap, gdb and perf simply by recompiling the source with a recent enough GCC. The DTRACE markers are source compatible, so with that simple tweak you can now easily use them from any of the observability, debugging, tracing or profiling tools on GNU/Linux that can read SDT probes in ELF files. That is in general the case with any dtrace SDT probes. Just rebuild the sources with /usr/include/sys/sdt.h installed (and maybe a small configure check or define added) and tada! > > So at this point I'm wondering a few things: > > > > * Is there any good reason why BSD (and Linux) support for JSDT has > > not been ported or implemented? > > * What is the current thought on the future of DTrace support in OpenJDK? > > > > Our steps going forward, if we want to support JRuby dtrace probes on > > platforms other than Solaris, will be to either implement or coerce > > someone into implementing BSD/Linux DTrace JSDT support in OpenJDK or > > to use a third-party library and our own JNI bindings. I'd prefer to > > just get it working in OpenJDK proper. > > > > Thoughts? JSDT is a general framework for providing some tracing support in hotspot/openjdk introduced with 1.7. One provider is based on dtrace (solaris specific atm). I am not sure there is "official" documentation for it. It openjdk specific and lives in the private package com.sun.tracing.*. One idea is to just implement a tracing provider through Byteman. https://www.jboss.org/byteman That would give you a "pure java" tracing factory, which might interface easier with JRuby. As a bonus systemtap comes with some Byteman hooks to use Byteman rewrite rules to make trace points visible as SDT probes in a JNI library that can be observed by systemtap. https://lukasberk.wordpress.com/2013/06/20/probing-java-methods-with-systemtap/ The JSDT DTraceProviderFactory makes use of a feature that is dtrace/kernel specific. So it takes a bit more to implement it than just a recompile as is the case with normal SDT markers. It opens a special file /dev/dtrace/helper with which it does a secret handshake to tell dtrace on solaris where the hotspot JIT created markers in the code and the name it wants these probes points to have. This is communicated by creating a fake SUNW_DOF ELF section that is specific to how dtrace handles SDT probes. There doesn't seem to be much documentation on these dtrace specific special file helper and the SUNW_DOF ELF section. But the concept is clear. For GNU/Linux we currently don't have such a registration service for new SDT probes. So we should create one. Probably based on the stapsdt NOTEs that the various tools already use to detect SDT probes. I am not sure it needs to be a special linux kernel device or proc file. Maybe it can be a generic daemon (using dbus to register and query which probe providers are currently active maybe?). It just has to make sure things are synchronized so tools won't place a probe at a point that has been removed already. And we should make sure that all tools (gdb, systemtap, perf) can query it and agree on the format. The only issue I see is that the current uprobes support in the linux kernel is inode/offset based, which might make it a little harder to probe these none-file-backed JIT code pages (except for gdb of course which doesn't use uprobes directly). One idea would be to add support for such a new service to libusdt https://github.com/chrisa/libusdt and reimplement the hotspot JSDT provider in terms of libusdt to make everything magically work on all systems. And as added bonus make these more dynamic SDT probes also magically work everywhere by just a simple recompile like normal (DTRACE) SDT probes. Cheers, Mark
