Hi Vincent, On Wed, May 07, 2014 at 06:35:06PM +0200, Vincent Bernat wrote: > ??? 7 mai 2014 11:15 +0200, Willy Tarreau <[email protected]> : > > >> haproxy does not include DTrace probes by any chance right? :) > > > > No, and I have no idea how this works either. But if you feel like it > > can provide some value and be done without too much effort, feel free > > to try :-) > > Here is a proof of concept. To test, use `make TARGET=linux2628 > USE_DTRACE=1`. On Linux, you need systemtap-sdt-dev or something like > that. Then, there is a quick example in example/haproxy.stp.
Interesting, but just for my understanding, what does it provide beyond building with "TRACE=1" where the compiler dumps *all* function calls, and not only those that were instrumented ? I'm asking because I never used dtrace, so I'm totally ignorant here. > You can try > it like this: > > #+begin_src sh > sudo stap ./examples/haproxy.stp > #+end_src > > It is possible to convert the probes.d to a tapset (which is a recipe > for systemtap) to be able to name arguments and convert them in the > appropriate type. I am using this AWK script: > > https://github.com/vincentbernat/lldpd/blob/master/src/daemon/dtrace2systemtap.awk > > Only works with simple probes. > > For dtrace, this would be something like that but I cannot test right > now: > > #+begin_src dtrace > haproxy$target:::frontend_accept > { > printf("Frontend %s accepted a connection", copyinstr(arg0)); > } > #+end_src > > The trick with those tracepoints is that they are just NOOP until you > enable them. So, even when someone compiles dtrace support, they will > not have any performance impact until trying to use the tracepoints. Well, they will at least have the performance impact of the "if" which disables them and the inflated/reordered functions I guess! So at least we have to be reasonable not to put them everywhere (eg: not in the polling loops nor in the scheduler). > While the probe arguments can be anything, it is simpler to only keep > simple types like null-terminated strings or int. Otherwise, they are > difficult to exploit. If you put struct, without the debug symbols, the > data is not exploitable. > > Now, all the hard work is to put trace points everywhere. That's where gcc does the stuff free of charge in fact. I still tend to be cautious about what the debugging code becomes over time, because we had this twice, once with the DPRINTF() macro which was never up to date, and once with the http_silent_debug() macro which became so unbalanced over time that I recently totally removed it. > A good target is where stuff are logged. Yeah that's a good idea. > But they can also be put in places where logs > would be too verbose. I currently don't have interest in doing that but > if someone is willing too, it is only a matter of defining the probes in > probes.d and placing them in the C code. This is really nifty to debug > stuff in production. However, I think that people interested in that can > also use debug symbols to place probe at any place they want to. GCC is > now better at providing debug symbols which work on optimized > executables. Ubuntu is providing debug symbols for almost > everything. Tracepoints are still interesting as they can be listed and > they are hand-picked. That was the principle of the http_silent_debug() in fact. Just to know where we passed, in which order at a low cost. But I think I failed at it by trying to maintain this code stable, while in practice we probably only need something properly instrumented to easily add new tracepoints when needed. Maybe your patch can be a nice step forward in that direction, I have no idea. It's not intrusive, that's possibly something we can merge and see if it is quickly adopted or not. Regards, Willy
