Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 18 November 2013 12:25, Sandeepa Prabhu wrote: >>> This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) >>> which places watchpoint for bothe read/write. >>> Atleast watchpt should have triggered for Read right? I also tried >>> with othe functions like do_fork, vfs_read etc but no hit. >> >> You'd need to place something for exec if you want to see anything on the >> instruction side. A read by the instruction fetcher does not trigger a read >> watchpoint on ARM. > Hmm, then watchpoint cannot not be placed on text address (instruction > fetch path) right? > (Sorry I did not check debug spec fully for watchpoint/hw breakpoints, > I should do that) > >> >>> >> 2. Placing text breakpoint (modified sample module with attr.bp_type >>> >> = HW_BREAKPOINT_X) upon vfs_symlink, and run "ln -s /dev/null >>> >> /tmp/foo". This time, breakpoint hit but exception is re-cursing >>> >> infinitely! >>> > >>> > The problem here is that we expect the overflow handler to deal with the >>> > stepping (like GDB does via ptrace). If you don't register a handler, the >>> > kernel will do the step (like you would get if you used perf stat -e >>> > mem:0x:x). >>> [This test was done on upstream branch, without kprobes patches.] >>> Hmm, then this is expected with test breakpoint right? is this >>> handling to be done by perf and ptrace? >> >> perf stat doesn't register an overflow handler, so the hw_breakpoint >> backend will handle the step. ptrace registers a handler which sends a >> SIGTRAP to the debugger (e.g. gdb), which handles the step manually >> (probably using a PTRACE_SINGLESTEP request). >> >>> I did not see arm64 support in linux/tools/perf/, there are multiple >>> patches in mailing list though. Are you aware of any version of perf >>> that work with arm64? >> >> The perf tool should work fine on arm64 using mainline. Are you seeing >> problems? > Hmm basically perf is working and I can run tests now. Hi Will, Okay. I can see what you meant with respect to D-flag unmasking, on original patchset(v2). :) a. Placed a kprobe on vfs_read - works fine independently. b. Placed hw_breakpoint on vfs_read using perf (perf record -e mem:0xffc000134ba4:x -a -- sleep 10) and it works fine independently. c. Now, a+b, first placed kprobe on vfs_read and then ran perf record event on vfs_read (hw breakpoint). Now, seeing that kprobe single step is never complete/never disabled!, so debug exception would generate forever! (Continuously printing "Unexpected kernel single-step exception at EL1") kprobe pre_handler: p->addr = 0xffc000134ba4 reenter_dbg: test API invoked kprobe post_handler: p->addr = 0xffc000134ba4 fmp/fo^[[5Dreenter_dbg: test API invoked kprobe pre_handler: p->addr = 0xffc000134ba4 Unexpected kernel single-step exception at EL1 Unexpected kernel single-step exception at EL1 Once I change the location of D-flag manipulation (your suggestion) and run enough tests, I would come back with more details and inputs. Thanks, Sandeepa >> >> Will >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 18 November 2013 12:25, Sandeepa Prabhu sandeepa.pra...@linaro.org wrote: This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) which places watchpoint for bothe read/write. Atleast watchpt should have triggered for Read right? I also tried with othe functions like do_fork, vfs_read etc but no hit. You'd need to place something for exec if you want to see anything on the instruction side. A read by the instruction fetcher does not trigger a read watchpoint on ARM. Hmm, then watchpoint cannot not be placed on text address (instruction fetch path) right? (Sorry I did not check debug spec fully for watchpoint/hw breakpoints, I should do that) 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run ln -s /dev/null /tmp/foo. This time, breakpoint hit but exception is re-cursing infinitely! The problem here is that we expect the overflow handler to deal with the stepping (like GDB does via ptrace). If you don't register a handler, the kernel will do the step (like you would get if you used perf stat -e mem:0x:x). [This test was done on upstream branch, without kprobes patches.] Hmm, then this is expected with test breakpoint right? is this handling to be done by perf and ptrace? perf stat doesn't register an overflow handler, so the hw_breakpoint backend will handle the step. ptrace registers a handler which sends a SIGTRAP to the debugger (e.g. gdb), which handles the step manually (probably using a PTRACE_SINGLESTEP request). I did not see arm64 support in linux/tools/perf/, there are multiple patches in mailing list though. Are you aware of any version of perf that work with arm64? The perf tool should work fine on arm64 using mainline. Are you seeing problems? Hmm basically perf is working and I can run tests now. Hi Will, Okay. I can see what you meant with respect to D-flag unmasking, on original patchset(v2). :) a. Placed a kprobe on vfs_read - works fine independently. b. Placed hw_breakpoint on vfs_read using perf (perf record -e mem:0xffc000134ba4:x -a -- sleep 10) and it works fine independently. c. Now, a+b, first placed kprobe on vfs_read and then ran perf record event on vfs_read (hw breakpoint). Now, seeing that kprobe single step is never complete/never disabled!, so debug exception would generate forever! (Continuously printing Unexpected kernel single-step exception at EL1) kprobe pre_handler: p-addr = 0xffc000134ba4 reenter_dbg: test API invoked kprobe post_handler: p-addr = 0xffc000134ba4 fmp/fo^[[5Dreenter_dbg: test API invoked kprobe pre_handler: p-addr = 0xffc000134ba4 Unexpected kernel single-step exception at EL1 Unexpected kernel single-step exception at EL1 Once I change the location of D-flag manipulation (your suggestion) and run enough tests, I would come back with more details and inputs. Thanks, Sandeepa Will -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
>> This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) >> which places watchpoint for bothe read/write. >> Atleast watchpt should have triggered for Read right? I also tried >> with othe functions like do_fork, vfs_read etc but no hit. > > You'd need to place something for exec if you want to see anything on the > instruction side. A read by the instruction fetcher does not trigger a read > watchpoint on ARM. Hmm, then watchpoint cannot not be placed on text address (instruction fetch path) right? (Sorry I did not check debug spec fully for watchpoint/hw breakpoints, I should do that) > >> >> 2. Placing text breakpoint (modified sample module with attr.bp_type >> >> = HW_BREAKPOINT_X) upon vfs_symlink, and run "ln -s /dev/null >> >> /tmp/foo". This time, breakpoint hit but exception is re-cursing >> >> infinitely! >> > >> > The problem here is that we expect the overflow handler to deal with the >> > stepping (like GDB does via ptrace). If you don't register a handler, the >> > kernel will do the step (like you would get if you used perf stat -e >> > mem:0x:x). >> [This test was done on upstream branch, without kprobes patches.] >> Hmm, then this is expected with test breakpoint right? is this >> handling to be done by perf and ptrace? > > perf stat doesn't register an overflow handler, so the hw_breakpoint > backend will handle the step. ptrace registers a handler which sends a > SIGTRAP to the debugger (e.g. gdb), which handles the step manually > (probably using a PTRACE_SINGLESTEP request). > >> I did not see arm64 support in linux/tools/perf/, there are multiple >> patches in mailing list though. Are you aware of any version of perf >> that work with arm64? > > The perf tool should work fine on arm64 using mainline. Are you seeing > problems? Hmm basically perf is working and I can run tests now. > > Will > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) which places watchpoint for bothe read/write. Atleast watchpt should have triggered for Read right? I also tried with othe functions like do_fork, vfs_read etc but no hit. You'd need to place something for exec if you want to see anything on the instruction side. A read by the instruction fetcher does not trigger a read watchpoint on ARM. Hmm, then watchpoint cannot not be placed on text address (instruction fetch path) right? (Sorry I did not check debug spec fully for watchpoint/hw breakpoints, I should do that) 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run ln -s /dev/null /tmp/foo. This time, breakpoint hit but exception is re-cursing infinitely! The problem here is that we expect the overflow handler to deal with the stepping (like GDB does via ptrace). If you don't register a handler, the kernel will do the step (like you would get if you used perf stat -e mem:0x:x). [This test was done on upstream branch, without kprobes patches.] Hmm, then this is expected with test breakpoint right? is this handling to be done by perf and ptrace? perf stat doesn't register an overflow handler, so the hw_breakpoint backend will handle the step. ptrace registers a handler which sends a SIGTRAP to the debugger (e.g. gdb), which handles the step manually (probably using a PTRACE_SINGLESTEP request). I did not see arm64 support in linux/tools/perf/, there are multiple patches in mailing list though. Are you aware of any version of perf that work with arm64? The perf tool should work fine on arm64 using mainline. Are you seeing problems? Hmm basically perf is working and I can run tests now. Will -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On Wed, Nov 13, 2013 at 03:55:42PM +, Sandeepa Prabhu wrote: > On 13 November 2013 20:01, Will Deacon wrote: > > On Wed, Nov 13, 2013 at 06:55:33AM +, Sandeepa Prabhu wrote: > >> 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | > >> HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not > >> triggering at all. > > > > vfs_symlink is a function. Why would you expect to write it? > This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) > which places watchpoint for bothe read/write. > Atleast watchpt should have triggered for Read right? I also tried > with othe functions like do_fork, vfs_read etc but no hit. You'd need to place something for exec if you want to see anything on the instruction side. A read by the instruction fetcher does not trigger a read watchpoint on ARM. > >> 2. Placing text breakpoint (modified sample module with attr.bp_type > >> = HW_BREAKPOINT_X) upon vfs_symlink, and run "ln -s /dev/null > >> /tmp/foo". This time, breakpoint hit but exception is re-cursing > >> infinitely! > > > > The problem here is that we expect the overflow handler to deal with the > > stepping (like GDB does via ptrace). If you don't register a handler, the > > kernel will do the step (like you would get if you used perf stat -e > > mem:0x:x). > [This test was done on upstream branch, without kprobes patches.] > Hmm, then this is expected with test breakpoint right? is this > handling to be done by perf and ptrace? perf stat doesn't register an overflow handler, so the hw_breakpoint backend will handle the step. ptrace registers a handler which sends a SIGTRAP to the debugger (e.g. gdb), which handles the step manually (probably using a PTRACE_SINGLESTEP request). > I did not see arm64 support in linux/tools/perf/, there are multiple > patches in mailing list though. Are you aware of any version of perf > that work with arm64? The perf tool should work fine on arm64 using mainline. Are you seeing problems? Will -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On Wed, Nov 13, 2013 at 03:55:42PM +, Sandeepa Prabhu wrote: On 13 November 2013 20:01, Will Deacon will.dea...@arm.com wrote: On Wed, Nov 13, 2013 at 06:55:33AM +, Sandeepa Prabhu wrote: 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not triggering at all. vfs_symlink is a function. Why would you expect to write it? This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) which places watchpoint for bothe read/write. Atleast watchpt should have triggered for Read right? I also tried with othe functions like do_fork, vfs_read etc but no hit. You'd need to place something for exec if you want to see anything on the instruction side. A read by the instruction fetcher does not trigger a read watchpoint on ARM. 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run ln -s /dev/null /tmp/foo. This time, breakpoint hit but exception is re-cursing infinitely! The problem here is that we expect the overflow handler to deal with the stepping (like GDB does via ptrace). If you don't register a handler, the kernel will do the step (like you would get if you used perf stat -e mem:0x:x). [This test was done on upstream branch, without kprobes patches.] Hmm, then this is expected with test breakpoint right? is this handling to be done by perf and ptrace? perf stat doesn't register an overflow handler, so the hw_breakpoint backend will handle the step. ptrace registers a handler which sends a SIGTRAP to the debugger (e.g. gdb), which handles the step manually (probably using a PTRACE_SINGLESTEP request). I did not see arm64 support in linux/tools/perf/, there are multiple patches in mailing list though. Are you aware of any version of perf that work with arm64? The perf tool should work fine on arm64 using mainline. Are you seeing problems? Will -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/13 1:59), Steven Rostedt wrote: > On Tue, 12 Nov 2013 16:25:26 +0530 > Sandeepa Prabhu wrote: > > >>> >>> BTW, I'm currently trying a general housecleaning of __kprobes >>> annotations. It may also have impact on your patch. >>> https://lkml.org/lkml/2013/11/8/187 >> Hmm, we can help testing your patchset on arm64 platforms. Also have >> many doubts on the changes you are working [blacklisting probes etc] >> >> Basically I had tried placing kprobe on memcpy() and the model hung >> (insmod never returned back!). Fast-model I have does not have option >> of any debug so no clue what happened!. >> memcpy() is low-level call being used internally within kprobes, so >> probably we cannot handle probe on that routine, but then how to make >> sure all such API are rejected by kprobe sub-system ? > > Working on ports of ftrace, I found that many of the functions in lib/ > are used by several locations that just can't be traced, due to how > low level they are. I just simply blacklisted the entire lib/ > directory (See the top of lib/Makefile) > > I wonder if there's an easy way to blacklist entire directories from > being used by kprobes too. Or at least do it by a file per file basis. Hm, perhaps we can do some magic in post-build script as kallsyms does. 1) make an object file 2) extract symbols from the file 3) put the symbols into data section as a list of strings 4) analyze the list at boot (init) time by using kallsyms how about this? :) Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 13 November 2013 20:01, Will Deacon wrote: > On Wed, Nov 13, 2013 at 06:55:33AM +, Sandeepa Prabhu wrote: >> >>> I'm unsure about arm64's debug feature behavior, what does happen when >> >>> it performs a single-step on sw-breakpoint? >> >>> >> Sandeepa: I think you need to retry Masami's test on the arm64 model, >> since >> I'm fairly sure it won't work as expected without some additional code. >> >>> >> >>> OK, anyway, for testing same one, we need to port ftrace first. So the >> >>> next >> > >> > Sorry for confusion, s/next/fallback is what I meant. Making a kprobe >> > module >> > can be done without ftrace port. >> > >> >>> plan is to make a kprobe module to put a probe (which just printk >> >>> something) >> >>> on a specific function (e.g. vfs_symlink), and run perf record with >> >>> hw-breakpoint as below >> >>> >> >>> $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo >> >>> >> >>> Note that 0xXX is the address of vfs_symlink. >> >>> >> >>> After that, you can see the message in dmesg and also check the perf >> >>> result >> >>> with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry >> >>> if >> >>> it works) >> Hi Will, Masami, >> >> I am not sure of 'perf' right now (my minimal rootfs doesn't have) but >> I tried to test hardware breakpoints using sample modules >> "samples/hw_breakpoint/" on arm64 upstream branch. This should use >> same kernel api as perf I believe. >> >> 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | >> HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not >> triggering at all. > > vfs_symlink is a function. Why would you expect to write it? This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) which places watchpoint for bothe read/write. Atleast watchpt should have triggered for Read right? I also tried with othe functions like do_fork, vfs_read etc but no hit. > >> 2. Placing text breakpoint (modified sample module with attr.bp_type >> = HW_BREAKPOINT_X) upon vfs_symlink, and run "ln -s /dev/null >> /tmp/foo". This time, breakpoint hit but exception is re-cursing >> infinitely! > > The problem here is that we expect the overflow handler to deal with the > stepping (like GDB does via ptrace). If you don't register a handler, the > kernel will do the step (like you would get if you used perf stat -e > mem:0x:x). [This test was done on upstream branch, without kprobes patches.] Hmm, then this is expected with test breakpoint right? is this handling to be done by perf and ptrace? I did not see arm64 support in linux/tools/perf/, there are multiple patches in mailing list though. Are you aware of any version of perf that work with arm64? ~Sandeepa > > Will > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On Wed, Nov 13, 2013 at 06:55:33AM +, Sandeepa Prabhu wrote: > >>> I'm unsure about arm64's debug feature behavior, what does happen when > >>> it performs a single-step on sw-breakpoint? > >>> > Sandeepa: I think you need to retry Masami's test on the arm64 model, > since > I'm fairly sure it won't work as expected without some additional code. > >>> > >>> OK, anyway, for testing same one, we need to port ftrace first. So the > >>> next > > > > Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module > > can be done without ftrace port. > > > >>> plan is to make a kprobe module to put a probe (which just printk > >>> something) > >>> on a specific function (e.g. vfs_symlink), and run perf record with > >>> hw-breakpoint as below > >>> > >>> $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo > >>> > >>> Note that 0xXX is the address of vfs_symlink. > >>> > >>> After that, you can see the message in dmesg and also check the perf > >>> result > >>> with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if > >>> it works) > Hi Will, Masami, > > I am not sure of 'perf' right now (my minimal rootfs doesn't have) but > I tried to test hardware breakpoints using sample modules > "samples/hw_breakpoint/" on arm64 upstream branch. This should use > same kernel api as perf I believe. > > 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | > HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not > triggering at all. vfs_symlink is a function. Why would you expect to write it? > 2. Placing text breakpoint (modified sample module with attr.bp_type > = HW_BREAKPOINT_X) upon vfs_symlink, and run "ln -s /dev/null > /tmp/foo". This time, breakpoint hit but exception is re-cursing > infinitely! The problem here is that we expect the overflow handler to deal with the stepping (like GDB does via ptrace). If you don't register a handler, the kernel will do the step (like you would get if you used perf stat -e mem:0x:x). Will -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/13 15:55), Sandeepa Prabhu wrote: I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? > Sandeepa: I think you need to retry Masami's test on the arm64 model, > since > I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next >> >> Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module >> can be done without ftrace port. >> plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if it works) > Hi Will, Masami, > > I am not sure of 'perf' right now (my minimal rootfs doesn't have) but > I tried to test hardware breakpoints using sample modules > "samples/hw_breakpoint/" on arm64 upstream branch. This should use > same kernel api as perf I believe. > > 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | > HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not > triggering at all. > 2. Placing text breakpoint (modified sample module with attr.bp_type > = HW_BREAKPOINT_X) upon vfs_symlink, and run "ln -s /dev/null > /tmp/foo". This time, breakpoint hit but exception is re-cursing > infinitely! Did you this without kprobes? If so, the hw_breakpoint porting on arm64 may have a bug. > I have attached the kernel logs for reference. So wanted to check if > hw breakpoint/watch-points are working on the upstream branch? Has it > been tested recently with sample modules or perf/ptrace? I've tested on x86 with the latest tip-tree kernel and it worked. Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/13 15:55), Sandeepa Prabhu wrote: I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Hi Will, Masami, I am not sure of 'perf' right now (my minimal rootfs doesn't have) but I tried to test hardware breakpoints using sample modules samples/hw_breakpoint/ on arm64 upstream branch. This should use same kernel api as perf I believe. 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not triggering at all. 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run ln -s /dev/null /tmp/foo. This time, breakpoint hit but exception is re-cursing infinitely! Did you this without kprobes? If so, the hw_breakpoint porting on arm64 may have a bug. I have attached the kernel logs for reference. So wanted to check if hw breakpoint/watch-points are working on the upstream branch? Has it been tested recently with sample modules or perf/ptrace? I've tested on x86 with the latest tip-tree kernel and it worked. Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On Wed, Nov 13, 2013 at 06:55:33AM +, Sandeepa Prabhu wrote: I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Hi Will, Masami, I am not sure of 'perf' right now (my minimal rootfs doesn't have) but I tried to test hardware breakpoints using sample modules samples/hw_breakpoint/ on arm64 upstream branch. This should use same kernel api as perf I believe. 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not triggering at all. vfs_symlink is a function. Why would you expect to write it? 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run ln -s /dev/null /tmp/foo. This time, breakpoint hit but exception is re-cursing infinitely! The problem here is that we expect the overflow handler to deal with the stepping (like GDB does via ptrace). If you don't register a handler, the kernel will do the step (like you would get if you used perf stat -e mem:0x:x). Will -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 13 November 2013 20:01, Will Deacon will.dea...@arm.com wrote: On Wed, Nov 13, 2013 at 06:55:33AM +, Sandeepa Prabhu wrote: I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Hi Will, Masami, I am not sure of 'perf' right now (my minimal rootfs doesn't have) but I tried to test hardware breakpoints using sample modules samples/hw_breakpoint/ on arm64 upstream branch. This should use same kernel api as perf I believe. 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not triggering at all. vfs_symlink is a function. Why would you expect to write it? This is generic test module (samples/hw_breakpoint/data_breakpoint.ko) which places watchpoint for bothe read/write. Atleast watchpt should have triggered for Read right? I also tried with othe functions like do_fork, vfs_read etc but no hit. 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run ln -s /dev/null /tmp/foo. This time, breakpoint hit but exception is re-cursing infinitely! The problem here is that we expect the overflow handler to deal with the stepping (like GDB does via ptrace). If you don't register a handler, the kernel will do the step (like you would get if you used perf stat -e mem:0x:x). [This test was done on upstream branch, without kprobes patches.] Hmm, then this is expected with test breakpoint right? is this handling to be done by perf and ptrace? I did not see arm64 support in linux/tools/perf/, there are multiple patches in mailing list though. Are you aware of any version of perf that work with arm64? ~Sandeepa Will -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/13 1:59), Steven Rostedt wrote: On Tue, 12 Nov 2013 16:25:26 +0530 Sandeepa Prabhu sandeepa.pra...@linaro.org wrote: BTW, I'm currently trying a general housecleaning of __kprobes annotations. It may also have impact on your patch. https://lkml.org/lkml/2013/11/8/187 Hmm, we can help testing your patchset on arm64 platforms. Also have many doubts on the changes you are working [blacklisting probes etc] Basically I had tried placing kprobe on memcpy() and the model hung (insmod never returned back!). Fast-model I have does not have option of any debug so no clue what happened!. memcpy() is low-level call being used internally within kprobes, so probably we cannot handle probe on that routine, but then how to make sure all such API are rejected by kprobe sub-system ? Working on ports of ftrace, I found that many of the functions in lib/ are used by several locations that just can't be traced, due to how low level they are. I just simply blacklisted the entire lib/ directory (See the top of lib/Makefile) I wonder if there's an easy way to blacklist entire directories from being used by kprobes too. Or at least do it by a file per file basis. Hm, perhaps we can do some magic in post-build script as kallsyms does. 1) make an object file 2) extract symbols from the file 3) put the symbols into data section as a list of strings 4) analyze the list at boot (init) time by using kallsyms how about this? :) Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 13 November 2013 12:25, Sandeepa Prabhu wrote: I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? > Sandeepa: I think you need to retry Masami's test on the arm64 model, > since > I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next >> >> Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module >> can be done without ftrace port. >> plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if it works) Hi Will, Masami, I am not sure of 'perf' right now (my minimal rootfs doesn't have) but I tried to test hardware breakpoints using sample modules "samples/hw_breakpoint/" on arm64 upstream branch. This should use same kernel api as perf I believe. 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not triggering at all. 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run "ln -s /dev/null /tmp/foo". This time, breakpoint hit but exception is re-cursing infinitely! I have attached the kernel logs for reference. So wanted to check if hw breakpoint/watch-points are working on the upstream branch? Has it been tested recently with sample modules or perf/ptrace? Thanks, Sandeepa Initializing cgroup subsys cpu Linux version 3.12.0-rc4+ (sandeepa@linaro-workstation) (gcc version 4.7.3 20130328 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.04-20130415 - Linaro GCC 2013.04) ) #24 SMP PREEMPT Wed Nov 13 12:04:03 IST 2013 CPU: AArch64 Processor [410fd0f0] revision 0 Machine: RTSM_VE_AEMv8A bootconsole [earlycon0] enabled PERCPU: Embedded 10 pages/cpu @ffc87ffa8000 s11776 r8192 d20992 u40960 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1034240 Kernel command line: console=ttyAMA0 root=/dev/mmcblk0p2 earlyprintk=pl011,0x1c09 consolelog=9 rw PID hash table entries: 4096 (order: 3, 32768 bytes) Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) software IO TLB [mem 0x8f800-0x8fc00] (64MB) mapped at [ffc87800-ffc87bff] Memory: 4058384K/4194304K available (3347K kernel code, 211K rwdata, 1164K rodata, 171K init, 154K bss, 135920K reserved) Virtual kernel memory layout: vmalloc : 0xff80 - 0xffbb (245759 MB) vmemmap : 0xffbc01c0 - 0xffbc1f80 ( 476 MB) modules : 0xffbffc00 - 0xffc0 (64 MB) memory : 0xffc0 - 0xffc88000 ( 34816 MB) .init : 0xffc0004e9000 - 0xffc000513e00 ( 172 kB) .text : 0xffc8 - 0xffc0004e8cf4 ( 4516 kB) .data : 0xffc000514000 - 0xffc000548e80 ( 212 kB) SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 Preemptible hierarchical RCU implementation. RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4. NR_IRQS:64 nr_irqs:64 0 Architected cp15 timer(s) running at 100.00MHz (phys). Console: colour dummy device 80x25 Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=100) pid_max: default: 32768 minimum: 301 Mount-cache hash table entries: 256 hw perfevents: enabled with arm/armv8-pmuv3 PMU driver, 9 counters available CPU1: Booted secondary processor CPU2: Booted secondary processor CPU3: Booted secondary processor Brought up 4 CPUs SMP: Total of 4 processors activated. devtmpfs: initialized atomic64 test passed NET: Registered protocol family 16 of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@3,/sysctl@02 vdso: 2 pages (1 code, 1 data) at base ffc000519000 hw-breakpoint: found 16 breakpoint and 16 watchpoint registers. Serial: AMBA PL011 UART driver 1c09.uart: ttyAMA0 at MMIO 0x1c09 (irq = 37, base_baud = 0) is a PL011 rev2 console [ttyAMA0] enabled, bootconsole disabled console [ttyAMA0] enabled, bootconsole disabled 1c0a.uart: ttyAMA1 at MMIO 0x1c0a (irq = 38, base_baud = 0) is a PL011 rev2 1c0b.uart: ttyAMA2 at MMIO 0x1c0b (irq = 39, base_baud = 0) is a PL011 rev2 1c0c.uart: ttyAMA3 at MMIO 0x1c0c (irq = 40, base_baud = 0) is a PL011 rev2 bio: create slab at 0 SCSI subsystem initialized Switched to clocksource arch_sys_counter NET: Registered protocol family 2 TCP established hash table
Re: Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/12 19:55), Sandeepa Prabhu wrote: > Thanks for steps, ARM64 ftrace patches are under review on arm mailing > list, I can contact the (linaro) developer implementing ftrace on > what's supported and then figure-out a way to test this concurrency of > kprobes breakpoint and hardware breakpoint. Would you mean this? :) http://www.spinics.net/lists/arm-kernel/msg278477.html Wow, it seems that this also has some works around instruction manipulation (and confusable filenames...) >>> I referred to: http://lwn.net/Articles/572323/ which is another >>> implementation and on LAKML >> >> OK, I'll check that (and looks good at a glance). >> By the way, I concern about Linaro guys who looks working a bit far >> from the LKML and original feature maintainers. Please contact them, >> I'm sure they don't bite your hand :) > Hmm sure, will convey to our developers/leads :-) Nice :) >> BTW, I'm currently trying a general housecleaning of __kprobes >> annotations. It may also have impact on your patch. >> https://lkml.org/lkml/2013/11/8/187 > Hmm, we can help testing your patchset on arm64 platforms. Also have > many doubts on the changes you are working [blacklisting probes etc] > > Basically I had tried placing kprobe on memcpy() and the model hung > (insmod never returned back!). Fast-model I have does not have option > of any debug so no clue what happened!. On x86, I can probe memcpy() safely. It depends on the kprobes (and breakpoint handling) implementation, and it could be found. > memcpy() is low-level call being used internally within kprobes, so > probably we cannot handle probe on that routine, but then how to make > sure all such API are rejected by kprobe sub-system ? I see, the blacklist still needs to be maintained. I periodically run a test for probing each function on my kernel, and if I found such problem, I added it on the blacklist. Currently I run the test only on x86, so perhaps, other arch does not have well tested yet. Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 12 November 2013 15:47, Masami Hiramatsu wrote: > (2013/11/12 17:44), Sandeepa Prabhu wrote: >> On 12 November 2013 12:57, Masami Hiramatsu >> wrote: >>> (2013/11/12 15:23), Sandeepa Prabhu wrote: >>> OK, I've ensured that the hw_breakpoint (from perf) can work >>> with kprobes (from ftrace) at the same address on x86. >>> So if arm64 already support hw_breakpoint on perf, kprobes should >>> work with it. >> >> Single-stepping on x86 is different to the step behaviour on arm64 >> afaik. On >> ARM, we have to manually remove the breakpoint, perform a single-step, >> then >> add the breakpoint again. If we re-enable debug exceptions in the kprobe >> handler, the step will complete early and we'll never step off the >> breakpoint. > > I'm unsure about arm64's debug feature behavior, what does happen when > it performs a single-step on sw-breakpoint? > >> Sandeepa: I think you need to retry Masami's test on the arm64 model, >> since >> I'm fairly sure it won't work as expected without some additional code. > > OK, anyway, for testing same one, we need to port ftrace first. So the > next >>> >>> Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module >>> can be done without ftrace port. >> Yes, got it, all my verification until now are done using sample >> modules only, looking out for perf (or some other mechanism: ptrace?) >> that uses v8 hw breakpoint. > > Yes, kprobe vs. perf and uprobe vs. ptrace :) > > > plan is to make a kprobe module to put a probe (which just printk > something) > on a specific function (e.g. vfs_symlink), and run perf record with > hw-breakpoint as below > > $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo > > Note that 0xXX is the address of vfs_symlink. > > After that, you can see the message in dmesg and also check the perf > result > with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if > it works) Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. >>> >>> Would you mean this? :) >>> http://www.spinics.net/lists/arm-kernel/msg278477.html >>> >>> Wow, it seems that this also has some works around instruction >>> manipulation (and confusable filenames...) >> I referred to: http://lwn.net/Articles/572323/ which is another >> implementation and on LAKML > > OK, I'll check that (and looks good at a glance). > By the way, I concern about Linaro guys who looks working a bit far > from the LKML and original feature maintainers. Please contact them, > I'm sure they don't bite your hand :) Hmm sure, will convey to our developers/leads :-) > > BTW, I'm currently trying a general housecleaning of __kprobes > annotations. It may also have impact on your patch. > https://lkml.org/lkml/2013/11/8/187 Hmm, we can help testing your patchset on arm64 platforms. Also have many doubts on the changes you are working [blacklisting probes etc] Basically I had tried placing kprobe on memcpy() and the model hung (insmod never returned back!). Fast-model I have does not have option of any debug so no clue what happened!. memcpy() is low-level call being used internally within kprobes, so probably we cannot handle probe on that routine, but then how to make sure all such API are rejected by kprobe sub-system ? ~Sandeepa > > Thank you, > > -- > Masami HIRAMATSU > IT Management Research Dept. Linux Technology Center > Hitachi, Ltd., Yokohama Research Laboratory > E-mail: masami.hiramatsu...@hitachi.com > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/12 17:44), Sandeepa Prabhu wrote: > On 12 November 2013 12:57, Masami Hiramatsu > wrote: >> (2013/11/12 15:23), Sandeepa Prabhu wrote: >> OK, I've ensured that the hw_breakpoint (from perf) can work >> with kprobes (from ftrace) at the same address on x86. >> So if arm64 already support hw_breakpoint on perf, kprobes should >> work with it. > > Single-stepping on x86 is different to the step behaviour on arm64 afaik. > On > ARM, we have to manually remove the breakpoint, perform a single-step, > then > add the breakpoint again. If we re-enable debug exceptions in the kprobe > handler, the step will complete early and we'll never step off the > breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? > Sandeepa: I think you need to retry Masami's test on the arm64 model, > since > I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next >> >> Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module >> can be done without ftrace port. > Yes, got it, all my verification until now are done using sample > modules only, looking out for perf (or some other mechanism: ptrace?) > that uses v8 hw breakpoint. Yes, kprobe vs. perf and uprobe vs. ptrace :) plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if it works) >>> Thanks for steps, ARM64 ftrace patches are under review on arm mailing >>> list, I can contact the (linaro) developer implementing ftrace on >>> what's supported and then figure-out a way to test this concurrency of >>> kprobes breakpoint and hardware breakpoint. >> >> Would you mean this? :) >> http://www.spinics.net/lists/arm-kernel/msg278477.html >> >> Wow, it seems that this also has some works around instruction >> manipulation (and confusable filenames...) > I referred to: http://lwn.net/Articles/572323/ which is another > implementation and on LAKML OK, I'll check that (and looks good at a glance). By the way, I concern about Linaro guys who looks working a bit far from the LKML and original feature maintainers. Please contact them, I'm sure they don't bite your hand :) BTW, I'm currently trying a general housecleaning of __kprobes annotations. It may also have impact on your patch. https://lkml.org/lkml/2013/11/8/187 Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 12 November 2013 12:57, Masami Hiramatsu wrote: > (2013/11/12 15:23), Sandeepa Prabhu wrote: > OK, I've ensured that the hw_breakpoint (from perf) can work > with kprobes (from ftrace) at the same address on x86. > So if arm64 already support hw_breakpoint on perf, kprobes should > work with it. Single-stepping on x86 is different to the step behaviour on arm64 afaik. On ARM, we have to manually remove the breakpoint, perform a single-step, then add the breakpoint again. If we re-enable debug exceptions in the kprobe handler, the step will complete early and we'll never step off the breakpoint. >>> >>> I'm unsure about arm64's debug feature behavior, what does happen when >>> it performs a single-step on sw-breakpoint? >>> Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. >>> >>> OK, anyway, for testing same one, we need to port ftrace first. So the next > > Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module > can be done without ftrace port. Yes, got it, all my verification until now are done using sample modules only, looking out for perf (or some other mechanism: ptrace?) that uses v8 hw breakpoint. > >>> plan is to make a kprobe module to put a probe (which just printk something) >>> on a specific function (e.g. vfs_symlink), and run perf record with >>> hw-breakpoint as below >>> >>> $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo >>> >>> Note that 0xXX is the address of vfs_symlink. >>> >>> After that, you can see the message in dmesg and also check the perf result >>> with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if >>> it works) >> Thanks for steps, ARM64 ftrace patches are under review on arm mailing >> list, I can contact the (linaro) developer implementing ftrace on >> what's supported and then figure-out a way to test this concurrency of >> kprobes breakpoint and hardware breakpoint. > > Would you mean this? :) > http://www.spinics.net/lists/arm-kernel/msg278477.html > > Wow, it seems that this also has some works around instruction > manipulation (and confusable filenames...) I referred to: http://lwn.net/Articles/572323/ which is another implementation and on LAKML > > Thank you, > > -- > Masami HIRAMATSU > IT Management Research Dept. Linux Technology Center > Hitachi, Ltd., Yokohama Research Laboratory > E-mail: masami.hiramatsu...@hitachi.com > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 12 November 2013 12:57, Masami Hiramatsu masami.hiramatsu...@hitachi.com wrote: (2013/11/12 15:23), Sandeepa Prabhu wrote: OK, I've ensured that the hw_breakpoint (from perf) can work with kprobes (from ftrace) at the same address on x86. So if arm64 already support hw_breakpoint on perf, kprobes should work with it. Single-stepping on x86 is different to the step behaviour on arm64 afaik. On ARM, we have to manually remove the breakpoint, perform a single-step, then add the breakpoint again. If we re-enable debug exceptions in the kprobe handler, the step will complete early and we'll never step off the breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. Yes, got it, all my verification until now are done using sample modules only, looking out for perf (or some other mechanism: ptrace?) that uses v8 hw breakpoint. plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. Would you mean this? :) http://www.spinics.net/lists/arm-kernel/msg278477.html Wow, it seems that this also has some works around instruction manipulation (and confusable filenames...) I referred to: http://lwn.net/Articles/572323/ which is another implementation and on LAKML Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/12 17:44), Sandeepa Prabhu wrote: On 12 November 2013 12:57, Masami Hiramatsu masami.hiramatsu...@hitachi.com wrote: (2013/11/12 15:23), Sandeepa Prabhu wrote: OK, I've ensured that the hw_breakpoint (from perf) can work with kprobes (from ftrace) at the same address on x86. So if arm64 already support hw_breakpoint on perf, kprobes should work with it. Single-stepping on x86 is different to the step behaviour on arm64 afaik. On ARM, we have to manually remove the breakpoint, perform a single-step, then add the breakpoint again. If we re-enable debug exceptions in the kprobe handler, the step will complete early and we'll never step off the breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. Yes, got it, all my verification until now are done using sample modules only, looking out for perf (or some other mechanism: ptrace?) that uses v8 hw breakpoint. Yes, kprobe vs. perf and uprobe vs. ptrace :) plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. Would you mean this? :) http://www.spinics.net/lists/arm-kernel/msg278477.html Wow, it seems that this also has some works around instruction manipulation (and confusable filenames...) I referred to: http://lwn.net/Articles/572323/ which is another implementation and on LAKML OK, I'll check that (and looks good at a glance). By the way, I concern about Linaro guys who looks working a bit far from the LKML and original feature maintainers. Please contact them, I'm sure they don't bite your hand :) BTW, I'm currently trying a general housecleaning of __kprobes annotations. It may also have impact on your patch. https://lkml.org/lkml/2013/11/8/187 Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 12 November 2013 15:47, Masami Hiramatsu masami.hiramatsu...@hitachi.com wrote: (2013/11/12 17:44), Sandeepa Prabhu wrote: On 12 November 2013 12:57, Masami Hiramatsu masami.hiramatsu...@hitachi.com wrote: (2013/11/12 15:23), Sandeepa Prabhu wrote: OK, I've ensured that the hw_breakpoint (from perf) can work with kprobes (from ftrace) at the same address on x86. So if arm64 already support hw_breakpoint on perf, kprobes should work with it. Single-stepping on x86 is different to the step behaviour on arm64 afaik. On ARM, we have to manually remove the breakpoint, perform a single-step, then add the breakpoint again. If we re-enable debug exceptions in the kprobe handler, the step will complete early and we'll never step off the breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. Yes, got it, all my verification until now are done using sample modules only, looking out for perf (or some other mechanism: ptrace?) that uses v8 hw breakpoint. Yes, kprobe vs. perf and uprobe vs. ptrace :) plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. Would you mean this? :) http://www.spinics.net/lists/arm-kernel/msg278477.html Wow, it seems that this also has some works around instruction manipulation (and confusable filenames...) I referred to: http://lwn.net/Articles/572323/ which is another implementation and on LAKML OK, I'll check that (and looks good at a glance). By the way, I concern about Linaro guys who looks working a bit far from the LKML and original feature maintainers. Please contact them, I'm sure they don't bite your hand :) Hmm sure, will convey to our developers/leads :-) BTW, I'm currently trying a general housecleaning of __kprobes annotations. It may also have impact on your patch. https://lkml.org/lkml/2013/11/8/187 Hmm, we can help testing your patchset on arm64 platforms. Also have many doubts on the changes you are working [blacklisting probes etc] Basically I had tried placing kprobe on memcpy() and the model hung (insmod never returned back!). Fast-model I have does not have option of any debug so no clue what happened!. memcpy() is low-level call being used internally within kprobes, so probably we cannot handle probe on that routine, but then how to make sure all such API are rejected by kprobe sub-system ? ~Sandeepa Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/12 19:55), Sandeepa Prabhu wrote: Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. Would you mean this? :) http://www.spinics.net/lists/arm-kernel/msg278477.html Wow, it seems that this also has some works around instruction manipulation (and confusable filenames...) I referred to: http://lwn.net/Articles/572323/ which is another implementation and on LAKML OK, I'll check that (and looks good at a glance). By the way, I concern about Linaro guys who looks working a bit far from the LKML and original feature maintainers. Please contact them, I'm sure they don't bite your hand :) Hmm sure, will convey to our developers/leads :-) Nice :) BTW, I'm currently trying a general housecleaning of __kprobes annotations. It may also have impact on your patch. https://lkml.org/lkml/2013/11/8/187 Hmm, we can help testing your patchset on arm64 platforms. Also have many doubts on the changes you are working [blacklisting probes etc] Basically I had tried placing kprobe on memcpy() and the model hung (insmod never returned back!). Fast-model I have does not have option of any debug so no clue what happened!. On x86, I can probe memcpy() safely. It depends on the kprobes (and breakpoint handling) implementation, and it could be found. memcpy() is low-level call being used internally within kprobes, so probably we cannot handle probe on that routine, but then how to make sure all such API are rejected by kprobe sub-system ? I see, the blacklist still needs to be maintained. I periodically run a test for probing each function on my kernel, and if I found such problem, I added it on the blacklist. Currently I run the test only on x86, so perhaps, other arch does not have well tested yet. Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 13 November 2013 12:25, Sandeepa Prabhu sandeepa.pra...@linaro.org wrote: I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Hi Will, Masami, I am not sure of 'perf' right now (my minimal rootfs doesn't have) but I tried to test hardware breakpoints using sample modules samples/hw_breakpoint/ on arm64 upstream branch. This should use same kernel api as perf I believe. 1. Placing watchpoint ( attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R) upon vfs_symlink symbol, but seems watch-point is not triggering at all. 2. Placing text breakpoint (modified sample module with attr.bp_type = HW_BREAKPOINT_X) upon vfs_symlink, and run ln -s /dev/null /tmp/foo. This time, breakpoint hit but exception is re-cursing infinitely! I have attached the kernel logs for reference. So wanted to check if hw breakpoint/watch-points are working on the upstream branch? Has it been tested recently with sample modules or perf/ptrace? Thanks, Sandeepa Initializing cgroup subsys cpu Linux version 3.12.0-rc4+ (sandeepa@linaro-workstation) (gcc version 4.7.3 20130328 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.04-20130415 - Linaro GCC 2013.04) ) #24 SMP PREEMPT Wed Nov 13 12:04:03 IST 2013 CPU: AArch64 Processor [410fd0f0] revision 0 Machine: RTSM_VE_AEMv8A bootconsole [earlycon0] enabled PERCPU: Embedded 10 pages/cpu @ffc87ffa8000 s11776 r8192 d20992 u40960 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1034240 Kernel command line: console=ttyAMA0 root=/dev/mmcblk0p2 earlyprintk=pl011,0x1c09 consolelog=9 rw PID hash table entries: 4096 (order: 3, 32768 bytes) Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) software IO TLB [mem 0x8f800-0x8fc00] (64MB) mapped at [ffc87800-ffc87bff] Memory: 4058384K/4194304K available (3347K kernel code, 211K rwdata, 1164K rodata, 171K init, 154K bss, 135920K reserved) Virtual kernel memory layout: vmalloc : 0xff80 - 0xffbb (245759 MB) vmemmap : 0xffbc01c0 - 0xffbc1f80 ( 476 MB) modules : 0xffbffc00 - 0xffc0 (64 MB) memory : 0xffc0 - 0xffc88000 ( 34816 MB) .init : 0xffc0004e9000 - 0xffc000513e00 ( 172 kB) .text : 0xffc8 - 0xffc0004e8cf4 ( 4516 kB) .data : 0xffc000514000 - 0xffc000548e80 ( 212 kB) SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 Preemptible hierarchical RCU implementation. RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4. NR_IRQS:64 nr_irqs:64 0 Architected cp15 timer(s) running at 100.00MHz (phys). Console: colour dummy device 80x25 Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=100) pid_max: default: 32768 minimum: 301 Mount-cache hash table entries: 256 hw perfevents: enabled with arm/armv8-pmuv3 PMU driver, 9 counters available CPU1: Booted secondary processor CPU2: Booted secondary processor CPU3: Booted secondary processor Brought up 4 CPUs SMP: Total of 4 processors activated. devtmpfs: initialized atomic64 test passed NET: Registered protocol family 16 of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@3,/sysctl@02 vdso: 2 pages (1 code, 1 data) at base ffc000519000 hw-breakpoint: found 16 breakpoint and 16 watchpoint registers. Serial: AMBA PL011 UART driver 1c09.uart: ttyAMA0 at MMIO 0x1c09 (irq = 37, base_baud = 0) is a PL011 rev2 console [ttyAMA0] enabled, bootconsole disabled console [ttyAMA0] enabled, bootconsole disabled 1c0a.uart: ttyAMA1 at MMIO 0x1c0a (irq = 38, base_baud = 0) is a PL011 rev2 1c0b.uart: ttyAMA2 at MMIO 0x1c0b (irq = 39, base_baud = 0) is a PL011 rev2 1c0c.uart: ttyAMA3 at MMIO 0x1c0c (irq = 40, base_baud = 0) is a PL011 rev2 bio: create slab bio-0 at 0 SCSI subsystem initialized Switched to clocksource arch_sys_counter NET: Registered protocol family 2 TCP established hash table entries: 32768 (order: 7, 524288 bytes) TCP bind hash table entries:
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/12 15:23), Sandeepa Prabhu wrote: OK, I've ensured that the hw_breakpoint (from perf) can work with kprobes (from ftrace) at the same address on x86. So if arm64 already support hw_breakpoint on perf, kprobes should work with it. >>> >>> Single-stepping on x86 is different to the step behaviour on arm64 afaik. On >>> ARM, we have to manually remove the breakpoint, perform a single-step, then >>> add the breakpoint again. If we re-enable debug exceptions in the kprobe >>> handler, the step will complete early and we'll never step off the >>> breakpoint. >> >> I'm unsure about arm64's debug feature behavior, what does happen when >> it performs a single-step on sw-breakpoint? >> >>> Sandeepa: I think you need to retry Masami's test on the arm64 model, since >>> I'm fairly sure it won't work as expected without some additional code. >> >> OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. >> plan is to make a kprobe module to put a probe (which just printk something) >> on a specific function (e.g. vfs_symlink), and run perf record with >> hw-breakpoint as below >> >> $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo >> >> Note that 0xXX is the address of vfs_symlink. >> >> After that, you can see the message in dmesg and also check the perf result >> with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if >> it works) > Thanks for steps, ARM64 ftrace patches are under review on arm mailing > list, I can contact the (linaro) developer implementing ftrace on > what's supported and then figure-out a way to test this concurrency of > kprobes breakpoint and hardware breakpoint. Would you mean this? :) http://www.spinics.net/lists/arm-kernel/msg278477.html Wow, it seems that this also has some works around instruction manipulation (and confusable filenames...) Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
>>> OK, I've ensured that the hw_breakpoint (from perf) can work >>> with kprobes (from ftrace) at the same address on x86. >>> So if arm64 already support hw_breakpoint on perf, kprobes should >>> work with it. >> >> Single-stepping on x86 is different to the step behaviour on arm64 afaik. On >> ARM, we have to manually remove the breakpoint, perform a single-step, then >> add the breakpoint again. If we re-enable debug exceptions in the kprobe >> handler, the step will complete early and we'll never step off the >> breakpoint. > > I'm unsure about arm64's debug feature behavior, what does happen when > it performs a single-step on sw-breakpoint? > >> Sandeepa: I think you need to retry Masami's test on the arm64 model, since >> I'm fairly sure it won't work as expected without some additional code. > > OK, anyway, for testing same one, we need to port ftrace first. So the next > plan is to make a kprobe module to put a probe (which just printk something) > on a specific function (e.g. vfs_symlink), and run perf record with > hw-breakpoint as below > > $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo > > Note that 0xXX is the address of vfs_symlink. > > After that, you can see the message in dmesg and also check the perf result > with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if > it works) Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. Thanks, Sandeepa > > Thank you, > > -- > Masami HIRAMATSU > IT Management Research Dept. Linux Technology Center > Hitachi, Ltd., Yokohama Research Laboratory > E-mail: masami.hiramatsu...@hitachi.com > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/11 19:58), Will Deacon wrote: > On Mon, Nov 11, 2013 at 10:51:52AM +, Masami Hiramatsu wrote: >> (2013/11/11 16:54), Masami Hiramatsu wrote: >> In fact, how do you avoid a race with hardware breakpoints? E.g., >> somebody >> places a hardware breakpoint on an instruction in the kernel for which >> kprobes has patched in a brk. We take the hardware breakpoint, disable >> the >> breakpoint and set up a single step before returning to the brk. The brk >> then traps, but we must take care not to disable single-step and/or >> unmask >> debug exceptions, because that will cause the hardware breakpoint code to >> re-arm its breakpoint before we've stepped off the brk instruction. > > Hmm, frankly to say, this kind of race issue is not seriously discussed > on x86 too, since kgdb is still a special tool (not used on the production > system). > I think under such situation kgdb operator must have full control of the > system, and he can (and has to) avoid such kind of race. Masami, Hmm I think in same lines, but not sure if we expect kprobes to be able to work fool-proof along with kgdb or hw breakpoints ? >>> >>> For hw breakpoint, yes, we finally get check each other to safely >>> use it even if one rejects the other one at some points(address). >>> Since the hw breakpoint is already open for normal user via perf, >>> we should do it. But the policy still needs to be discussed. >> >> OK, I've ensured that the hw_breakpoint (from perf) can work >> with kprobes (from ftrace) at the same address on x86. >> So if arm64 already support hw_breakpoint on perf, kprobes should >> work with it. > > Single-stepping on x86 is different to the step behaviour on arm64 afaik. On > ARM, we have to manually remove the breakpoint, perform a single-step, then > add the breakpoint again. If we re-enable debug exceptions in the kprobe > handler, the step will complete early and we'll never step off the > breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? > Sandeepa: I think you need to retry Masami's test on the arm64 model, since > I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e "mem:0xXX:k" ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with "sudo perf script --dump" (you can find a PERF_RECORD_SAMPLE entry if it works) Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/11 19:58), Will Deacon wrote: On Mon, Nov 11, 2013 at 10:51:52AM +, Masami Hiramatsu wrote: (2013/11/11 16:54), Masami Hiramatsu wrote: In fact, how do you avoid a race with hardware breakpoints? E.g., somebody places a hardware breakpoint on an instruction in the kernel for which kprobes has patched in a brk. We take the hardware breakpoint, disable the breakpoint and set up a single step before returning to the brk. The brk then traps, but we must take care not to disable single-step and/or unmask debug exceptions, because that will cause the hardware breakpoint code to re-arm its breakpoint before we've stepped off the brk instruction. Hmm, frankly to say, this kind of race issue is not seriously discussed on x86 too, since kgdb is still a special tool (not used on the production system). I think under such situation kgdb operator must have full control of the system, and he can (and has to) avoid such kind of race. Masami, Hmm I think in same lines, but not sure if we expect kprobes to be able to work fool-proof along with kgdb or hw breakpoints ? For hw breakpoint, yes, we finally get check each other to safely use it even if one rejects the other one at some points(address). Since the hw breakpoint is already open for normal user via perf, we should do it. But the policy still needs to be discussed. OK, I've ensured that the hw_breakpoint (from perf) can work with kprobes (from ftrace) at the same address on x86. So if arm64 already support hw_breakpoint on perf, kprobes should work with it. Single-stepping on x86 is different to the step behaviour on arm64 afaik. On ARM, we have to manually remove the breakpoint, perform a single-step, then add the breakpoint again. If we re-enable debug exceptions in the kprobe handler, the step will complete early and we'll never step off the breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
OK, I've ensured that the hw_breakpoint (from perf) can work with kprobes (from ftrace) at the same address on x86. So if arm64 already support hw_breakpoint on perf, kprobes should work with it. Single-stepping on x86 is different to the step behaviour on arm64 afaik. On ARM, we have to manually remove the breakpoint, perform a single-step, then add the breakpoint again. If we re-enable debug exceptions in the kprobe handler, the step will complete early and we'll never step off the breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. Thanks, Sandeepa Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/12 15:23), Sandeepa Prabhu wrote: OK, I've ensured that the hw_breakpoint (from perf) can work with kprobes (from ftrace) at the same address on x86. So if arm64 already support hw_breakpoint on perf, kprobes should work with it. Single-stepping on x86 is different to the step behaviour on arm64 afaik. On ARM, we have to manually remove the breakpoint, perform a single-step, then add the breakpoint again. If we re-enable debug exceptions in the kprobe handler, the step will complete early and we'll never step off the breakpoint. I'm unsure about arm64's debug feature behavior, what does happen when it performs a single-step on sw-breakpoint? Sandeepa: I think you need to retry Masami's test on the arm64 model, since I'm fairly sure it won't work as expected without some additional code. OK, anyway, for testing same one, we need to port ftrace first. So the next Sorry for confusion, s/next/fallback is what I meant. Making a kprobe module can be done without ftrace port. plan is to make a kprobe module to put a probe (which just printk something) on a specific function (e.g. vfs_symlink), and run perf record with hw-breakpoint as below $ perf record -e mem:0xXX:k ln -s /dev/null /tmp/foo Note that 0xXX is the address of vfs_symlink. After that, you can see the message in dmesg and also check the perf result with sudo perf script --dump (you can find a PERF_RECORD_SAMPLE entry if it works) Thanks for steps, ARM64 ftrace patches are under review on arm mailing list, I can contact the (linaro) developer implementing ftrace on what's supported and then figure-out a way to test this concurrency of kprobes breakpoint and hardware breakpoint. Would you mean this? :) http://www.spinics.net/lists/arm-kernel/msg278477.html Wow, it seems that this also has some works around instruction manipulation (and confusable filenames...) Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 9 November 2013 14:40, Masami Hiramatsu wrote: > (2013/11/09 1:56), Will Deacon wrote: >> Hi Sandeepa, >> >> On Thu, Oct 17, 2013 at 12:17:47PM +0100, Sandeepa Prabhu wrote: >>> Add support for basic kernel probes(kprobes), jump probes (jprobes) >>> for ARM64. >> >> I think this series will conflict quite heavily with the jump_label series, >> since they both introduce some common instruction manipulation code. On the >> debug side, there will also be conflicts with the kgdb series, so it might >> make sense for us to merge those two first, then you can rebase on a stable >> branch from us. > > [...] > >> In fact, how do you avoid a race with hardware breakpoints? E.g., somebody >> places a hardware breakpoint on an instruction in the kernel for which >> kprobes has patched in a brk. We take the hardware breakpoint, disable the >> breakpoint and set up a single step before returning to the brk. The brk >> then traps, but we must take care not to disable single-step and/or unmask >> debug exceptions, because that will cause the hardware breakpoint code to >> re-arm its breakpoint before we've stepped off the brk instruction. > > Hmm, frankly to say, this kind of race issue is not seriously discussed > on x86 too, since kgdb is still a special tool (not used on the production > system). > I think under such situation kgdb operator must have full control of the > system, and he can (and has to) avoid such kind of race. Masami, Hmm I think in same lines, but not sure if we expect kprobes to be able to work fool-proof along with kgdb or hw breakpoints ? Thanks, Sandeepa > > Thank you, > > -- > Masami HIRAMATSU > IT Management Research Dept. Linux Technology Center > Hitachi, Ltd., Yokohama Research Laboratory > E-mail: masami.hiramatsu...@hitachi.com > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
On 9 November 2013 14:40, Masami Hiramatsu masami.hiramatsu...@hitachi.com wrote: (2013/11/09 1:56), Will Deacon wrote: Hi Sandeepa, On Thu, Oct 17, 2013 at 12:17:47PM +0100, Sandeepa Prabhu wrote: Add support for basic kernel probes(kprobes), jump probes (jprobes) for ARM64. I think this series will conflict quite heavily with the jump_label series, since they both introduce some common instruction manipulation code. On the debug side, there will also be conflicts with the kgdb series, so it might make sense for us to merge those two first, then you can rebase on a stable branch from us. [...] In fact, how do you avoid a race with hardware breakpoints? E.g., somebody places a hardware breakpoint on an instruction in the kernel for which kprobes has patched in a brk. We take the hardware breakpoint, disable the breakpoint and set up a single step before returning to the brk. The brk then traps, but we must take care not to disable single-step and/or unmask debug exceptions, because that will cause the hardware breakpoint code to re-arm its breakpoint before we've stepped off the brk instruction. Hmm, frankly to say, this kind of race issue is not seriously discussed on x86 too, since kgdb is still a special tool (not used on the production system). I think under such situation kgdb operator must have full control of the system, and he can (and has to) avoid such kind of race. Masami, Hmm I think in same lines, but not sure if we expect kprobes to be able to work fool-proof along with kgdb or hw breakpoints ? Thanks, Sandeepa Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/09 1:56), Will Deacon wrote: > Hi Sandeepa, > > On Thu, Oct 17, 2013 at 12:17:47PM +0100, Sandeepa Prabhu wrote: >> Add support for basic kernel probes(kprobes), jump probes (jprobes) >> for ARM64. > > I think this series will conflict quite heavily with the jump_label series, > since they both introduce some common instruction manipulation code. On the > debug side, there will also be conflicts with the kgdb series, so it might > make sense for us to merge those two first, then you can rebase on a stable > branch from us. [...] > In fact, how do you avoid a race with hardware breakpoints? E.g., somebody > places a hardware breakpoint on an instruction in the kernel for which > kprobes has patched in a brk. We take the hardware breakpoint, disable the > breakpoint and set up a single step before returning to the brk. The brk > then traps, but we must take care not to disable single-step and/or unmask > debug exceptions, because that will cause the hardware breakpoint code to > re-arm its breakpoint before we've stepped off the brk instruction. Hmm, frankly to say, this kind of race issue is not seriously discussed on x86 too, since kgdb is still a special tool (not used on the production system). I think under such situation kgdb operator must have full control of the system, and he can (and has to) avoid such kind of race. Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH RFC 2/6] arm64: Kprobes with single stepping support
(2013/11/09 1:56), Will Deacon wrote: Hi Sandeepa, On Thu, Oct 17, 2013 at 12:17:47PM +0100, Sandeepa Prabhu wrote: Add support for basic kernel probes(kprobes), jump probes (jprobes) for ARM64. I think this series will conflict quite heavily with the jump_label series, since they both introduce some common instruction manipulation code. On the debug side, there will also be conflicts with the kgdb series, so it might make sense for us to merge those two first, then you can rebase on a stable branch from us. [...] In fact, how do you avoid a race with hardware breakpoints? E.g., somebody places a hardware breakpoint on an instruction in the kernel for which kprobes has patched in a brk. We take the hardware breakpoint, disable the breakpoint and set up a single step before returning to the brk. The brk then traps, but we must take care not to disable single-step and/or unmask debug exceptions, because that will cause the hardware breakpoint code to re-arm its breakpoint before we've stepped off the brk instruction. Hmm, frankly to say, this kind of race issue is not seriously discussed on x86 too, since kgdb is still a special tool (not used on the production system). I think under such situation kgdb operator must have full control of the system, and he can (and has to) avoid such kind of race. Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
