Re: [lldb-dev] [RFC] Fast Conditional Breakpoints (FCB)
On 20/08/2019 00:11, Ismail Bennani wrote: On Aug 19, 2019, at 2:30 PM, Frédéric Riss wrote: On Aug 16, 2019, at 11:13 AM, Ismail Bennani via lldb-dev wrote: Hi Pavel, Thanks for all your feedbacks. I’ve been following the discussion closely and find your approach quite interesting. As Jim explained, I’m also trying to have a conditional breakpoint, that is able to stop a specific thread (name or id) when the condition expression evaluates to true. I feel like stacking up options with your approach would imply doing more context switches. But it’s definitely a better fallback mechanism than the current one. I’ll try to make a prototype to see the performance difference for both approaches. On Aug 15, 2019, at 10:10 AM, Pavel Labath wrote: Hello Ismail, and wellcome to LLDB. You have a very interesting (and not entirely trivial) project, and I wish you the best of luck in your work. I think this will be a very useful addition to lldb. It sounds like you have researched the problem very well, and the overall direction looks good to me. However, I do have some ideas suggestions about possible tweaks/improvements that I would like to hear your thoughts on. Please find my comments inline. On 14/08/2019 22:52, Ismail Bennani via lldb-dev wrote: Hi everyone, I’m Ismail, a compiler engineer intern at Apple. As a part of my internship, I'm adding Fast Conditional Breakpoints to LLDB, using code patching. Currently, the expressions that power conditional breakpoints are lowered to LLVM IR and LLDB knows how to interpret a subset of it. If that fails, the debugger JIT-compiles the expression (compiled once, and re-run on each breakpoint hit). In both cases LLDB must collect all program state used in the condition and pass it to the expression. The goal of my internship project is to make conditional breakpoints faster by: 1. Compiling the expression ahead-of-time, when setting the breakpoint and inject into the inferior memory only once. 2. Re-route the inferior execution flow to run the expression and check whether it needs to stop, in-process. This saves the cost of having to do the context switch between debugger and the inferior program (about 10 times) to compile and evaluate the condition. This feature is described on the [LLDB Project page](https://lldb.llvm.org/status/projects.html#use-the-jit-to-speed-up-conditional-breakpoint-evaluation). The goal would be to have it working for most languages and architectures supported by LLDB, however my original implementation will be for C-based languages targeting x86_64. It will be extended to AArch64 afterwards. Note the way my prototype is implemented makes it fully extensible for other languages and architectures. ## High Level Design Every time a breakpoint that holds a condition is hit, multiple context switches are needed in order to compile and evaluate the condition. First, the breakpoint is hit and the control is given to the debugger. That's where LLDB wraps the condition expression into a UserExpression that will get compiled and injected into the program memory. Another round-trip between the inferior and the LLDB is needed to run the compiled expression and extract the expression results that will tell LLDB to stop or not. To get rid of those context switches, we will evaluate the condition inside the program, and only stop when the condition is true. LLDB will achieve this by inserting a jump from the breakpoint address to a code section that will be allocated into the program memory. It will save the thread state, run the condition expression, restore the thread state and then execute the copied instruction(s) before jumping back to the regular program flow. Then we only trap and return control to LLDB when the condition is true. ## Implementation Details To be able to evaluate a breakpoint condition without interacting with the debugger, LLDB changes the inferior program execution flow by overwriting the instruction at which the breakpoint was set with a branching instruction. The original instruction(s) are copied to a memory stub allocated in the inferior program memory called the __Fast Conditional Breakpoint Trampoline__ or __FCBT__. The FCBT will allow us the re-route the program execution flow to check the condition in-process while preserving the original program behavior. This part is critical to setup Fast Conditional Breakpoints. ``` Inferior Binary Trampoline |.| +-+ |.| | | |.| +->+ Save RegisterContext | |.| | | | +-+ | +-+ | | | | | | Instruction | |
Re: [lldb-dev] [Release-testers] [9.0.0 Release] Release Candidate 2 is here
Hi Bernhard, On Sat, Aug 17, 2019 at 4:30 PM Bernhard Rosenkraenzer wrote: > The only serious issue we've run into so far is that clang crashes when > trying to build current firefox on x86_64. Will put up a bug report later. Did you get a chance to file that bug report yet? Thanks, Hans ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [cfe-dev] [Release-testers] [9.0.0 Release] Release Candidate 2 is here
On Sun, Aug 18, 2019 at 6:42 AM Marshall Clow wrote: > > > > On Fri, Aug 16, 2019 at 2:23 AM Sylvestre Ledru via cfe-dev > wrote: >> One severe regression on Debian on libc++ on mips* >> https://bugs.llvm.org/show_bug.cgi?id=43011 >> >> Simon found the regressor! >> >> Besides that, we look good. I will try to get a rebuild of the Debian >> archive with this version. > > > I'll have this fixed by Monday at the latest. > (probably tomorrow) Great! Please keep me posted on the bug. ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Release-testers] [9.0.0 Release] Release Candidate 2 is here
No, there should be no mismatch now. Maybe you ended up using a version of test-release.sh that doen't have the fixes? On Sun, Aug 18, 2019 at 6:47 PM Brian Cain wrote: > > When I built rv2 for SLES11 linux I saw the same phase 2/3 mismatch. Is that > expected? > > On Wed, Aug 14, 2019, 3:15 AM Hans Wennborg via Release-testers > wrote: >> >> Hello everyone, >> >> 9.0.0-rc2 was tagged yesterday from the release_90 branch at r368683. >> In the Git monorepo it's available as the llvmorg-9.0.0-rc2 tag. >> >> Source code and docs are available at https://prereleases.llvm.org/9.0.0/#rc2 >> >> Binaries will be added as they become available. >> >> The tag went in roughly one week behind schedule (see "Upcoming >> Releases" at https://llvm.org), but there are still two weeks left to >> the planned release date. >> >> Please file bug reports for any issues you find and mark them blocking >> https://llvm.org/PR42474 Please also look at the blockers and see if >> there's anything you can help with -- there are several bugs which >> lack traction at the moment. >> >> Release testers: please start your engines, run the script, share your >> results, and upload binaries. >> >> Many thanks, >> Hans >> ___ >> Release-testers mailing list >> release-test...@lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/release-testers ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [RFC] Fast Conditional Breakpoints (FCB)
It is great that you are looking at supporting these fast breakpoints but I am concerned about the instruction moving code along the same lines Pavel mentioned. Copying instructions from 1 location to another is fairly complicated even without considering the issue of jump targets and jump target detection makes it even harder. For reference, I implemented a similar system to do code shifting only on prologue instructions using LLVM what you might find useful for reference at https://github.com/google/gapid/tree/master/gapii/interceptor-lib/cc (Apache v2 license) in case you decide to go down this path. That system doesn't try to detect jump targets and only handles a small subset of the instructions but I think shows the general complexity. On X86_64 I think the number of instructions needs rewriting are relatively small as most of them aren't PC relative but for example on ARM where (almost) any instruction can take PC as a register it will be a monumental task that is very hard to test (I would expect AArch64 to be somewhere between X86_64 and ARM in terms of complexity due to PC relative instructions but no general purpose PC register). In my view this discussion leads to the question of how we trade performance for accuracy/reliability. We can easily gain a lot of performance by being a bit sloppy and assume that we can safely insert trampolines into the middle of the function but I would want my debugger to "never lie" or crash my program. Tamas On Tue, Aug 20, 2019 at 8:46 AM Pavel Labath via lldb-dev wrote: > > On 20/08/2019 00:11, Ismail Bennani wrote: > >> On Aug 19, 2019, at 2:30 PM, Frédéric Riss wrote: > >> > >> > >> > >>> On Aug 16, 2019, at 11:13 AM, Ismail Bennani via lldb-dev > >>> wrote: > >>> > >>> Hi Pavel, > >>> > >>> Thanks for all your feedbacks. > >>> > >>> I’ve been following the discussion closely and find your approach quite > >>> interesting. > >>> > >>> As Jim explained, I’m also trying to have a conditional breakpoint, that > >>> is able to stop a specific thread (name or id) when the condition > >>> expression evaluates to true. > >>> > >>> I feel like stacking up options with your approach would imply doing more > >>> context switches. > >>> But it’s definitely a better fallback mechanism than the current one. > >>> I’ll try to make a prototype to see the performance difference for both > >>> approaches. > >>> > >>> > On Aug 15, 2019, at 10:10 AM, Pavel Labath wrote: > > Hello Ismail, and wellcome to LLDB. You have a very interesting (and not > entirely trivial) project, and I wish you the best of luck in your work. > I think this will be a very useful addition to lldb. > > It sounds like you have researched the problem very well, and the > overall direction looks good to me. However, I do have some ideas > suggestions about possible tweaks/improvements that I would like to hear > your thoughts on. Please find my comments inline. > > On 14/08/2019 22:52, Ismail Bennani via lldb-dev wrote: > > Hi everyone, > > I’m Ismail, a compiler engineer intern at Apple. As a part of my > > internship, > > I'm adding Fast Conditional Breakpoints to LLDB, using code patching. > > Currently, the expressions that power conditional breakpoints are > > lowered > > to LLVM IR and LLDB knows how to interpret a subset of it. If that > > fails, > > the debugger JIT-compiles the expression (compiled once, and re-run on > > each > > breakpoint hit). In both cases LLDB must collect all program state used > > in > > the condition and pass it to the expression. > > The goal of my internship project is to make conditional breakpoints > > faster by: > > 1. Compiling the expression ahead-of-time, when setting the breakpoint > > and > >inject into the inferior memory only once. > > 2. Re-route the inferior execution flow to run the expression and check > > whether > >it needs to stop, in-process. > > This saves the cost of having to do the context switch between debugger > > and > > the inferior program (about 10 times) to compile and evaluate the > > condition. > > This feature is described on the [LLDB Project > > page](https://lldb.llvm.org/status/projects.html#use-the-jit-to-speed-up-conditional-breakpoint-evaluation). > > The goal would be to have it working for most languages and > > architectures > > supported by LLDB, however my original implementation will be for > > C-based > > languages targeting x86_64. It will be extended to AArch64 afterwards. > > Note the way my prototype is implemented makes it fully extensible for > > other > > languages and architectures. > > ## High Level Design > > Every time a breakpoint that holds a condition is hit, multiple context > > switches are needed in order to compile and evaluate the condition. > > Firs
Re: [lldb-dev] [Release-testers] [9.0.0 Release] Release Candidate 2 is here
Oh! Indeed, that was probably it. I'll try again. On Tue, Aug 20, 2019, 5:26 AM Hans Wennborg wrote: > No, there should be no mismatch now. Maybe you ended up using a > version of test-release.sh that doen't have the fixes? > > On Sun, Aug 18, 2019 at 6:47 PM Brian Cain wrote: > > > > When I built rv2 for SLES11 linux I saw the same phase 2/3 mismatch. Is > that expected? > > > > On Wed, Aug 14, 2019, 3:15 AM Hans Wennborg via Release-testers < > release-test...@lists.llvm.org> wrote: > >> > >> Hello everyone, > >> > >> 9.0.0-rc2 was tagged yesterday from the release_90 branch at r368683. > >> In the Git monorepo it's available as the llvmorg-9.0.0-rc2 tag. > >> > >> Source code and docs are available at > https://prereleases.llvm.org/9.0.0/#rc2 > >> > >> Binaries will be added as they become available. > >> > >> The tag went in roughly one week behind schedule (see "Upcoming > >> Releases" at https://llvm.org), but there are still two weeks left to > >> the planned release date. > >> > >> Please file bug reports for any issues you find and mark them blocking > >> https://llvm.org/PR42474 Please also look at the blockers and see if > >> there's anything you can help with -- there are several bugs which > >> lack traction at the moment. > >> > >> Release testers: please start your engines, run the script, share your > >> results, and upload binaries. > >> > >> Many thanks, > >> Hans > >> ___ > >> Release-testers mailing list > >> release-test...@lists.llvm.org > >> https://lists.llvm.org/cgi-bin/mailman/listinfo/release-testers > ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [RFC] Fast Conditional Breakpoints (FCB)
> On Aug 20, 2019, at 3:30 AM, Tamas Berghammer via lldb-dev > wrote: > > It is great that you are looking at supporting these fast breakpoints > but I am concerned about the instruction moving code along the same > lines Pavel mentioned. Copying instructions from 1 location to another > is fairly complicated even without considering the issue of jump > targets and jump target detection makes it even harder. > > For reference, I implemented a similar system to do code shifting only > on prologue instructions using LLVM what you might find useful for > reference at > https://github.com/google/gapid/tree/master/gapii/interceptor-lib/cc > (Apache v2 license) in case you decide to go down this path. > > That system doesn't try to detect jump targets and only handles a > small subset of the instructions but I think shows the general > complexity. On X86_64 I think the number of instructions needs > rewriting are relatively small as most of them aren't PC relative but > for example on ARM where (almost) any instruction can take PC as a > register it will be a monumental task that is very hard to test (I > would expect AArch64 to be somewhere between X86_64 and ARM in terms > of complexity due to PC relative instructions but no general purpose > PC register). > > In my view this discussion leads to the question of how we trade > performance for accuracy/reliability. We can easily gain a lot of > performance by being a bit sloppy and assume that we can safely insert > trampolines into the middle of the function but I would want my > debugger to "never lie" or crash my program. > While this can indeed be complicated, provided we are humble about our abilities, and back out on instructions we can't handle, we won't ever break your program. We'll just sometimes fail to set fast conditional breakpoints, and fall back to slow ones. We can announce this fact, and maybe even say "would it be okay if I moved this breakpoint two instructions north, then I could support fast conditions". So either the user can allow us to auto-adjust the location or pick another one by hand. We could even implement Pavel's suggestion as the fallback so we would still get some speedup. But as I said earlier in the thread, we're going to have to figure out how to do something like this if we really want to have "keep alive" threads in the presence of breakpoints. Since that seems a generally desirable feature, I think it's worth the effort to try to make this work. Jim > Tamas > > On Tue, Aug 20, 2019 at 8:46 AM Pavel Labath via lldb-dev > wrote: >> >> On 20/08/2019 00:11, Ismail Bennani wrote: On Aug 19, 2019, at 2:30 PM, Frédéric Riss wrote: > On Aug 16, 2019, at 11:13 AM, Ismail Bennani via lldb-dev > wrote: > > Hi Pavel, > > Thanks for all your feedbacks. > > I’ve been following the discussion closely and find your approach quite > interesting. > > As Jim explained, I’m also trying to have a conditional breakpoint, that > is able to stop a specific thread (name or id) when the condition > expression evaluates to true. > > I feel like stacking up options with your approach would imply doing more > context switches. > But it’s definitely a better fallback mechanism than the current one. > I’ll try to make a prototype to see the performance difference for both > approaches. > > >> On Aug 15, 2019, at 10:10 AM, Pavel Labath wrote: >> >> Hello Ismail, and wellcome to LLDB. You have a very interesting (and not >> entirely trivial) project, and I wish you the best of luck in your work. >> I think this will be a very useful addition to lldb. >> >> It sounds like you have researched the problem very well, and the >> overall direction looks good to me. However, I do have some ideas >> suggestions about possible tweaks/improvements that I would like to hear >> your thoughts on. Please find my comments inline. >> >> On 14/08/2019 22:52, Ismail Bennani via lldb-dev wrote: >>> Hi everyone, >>> I’m Ismail, a compiler engineer intern at Apple. As a part of my >>> internship, >>> I'm adding Fast Conditional Breakpoints to LLDB, using code patching. >>> Currently, the expressions that power conditional breakpoints are >>> lowered >>> to LLVM IR and LLDB knows how to interpret a subset of it. If that >>> fails, >>> the debugger JIT-compiles the expression (compiled once, and re-run on >>> each >>> breakpoint hit). In both cases LLDB must collect all program state used >>> in >>> the condition and pass it to the expression. >>> The goal of my internship project is to make conditional breakpoints >>> faster by: >>> 1. Compiling the expression ahead-of-time, when setting the breakpoint >>> and >>> inject into the inferior memory only once. >>> 2. Re-route the inferi
[lldb-dev] [Bug 22232] Conditional breakpoint test skipped for Windows
Hi, While looking at the LLDB test suite, I noticed that the test for conditional breakpoints (TestBreakpointConditions.py) is skipped on Windows. The test has an annotation `skipIfWindows` with a comment: `# Requires EE to support COFF on Windows (http://llvm.org/pr22232)` This bug was reported in 2015 and since then, I assume Windows gained support for expression evaluation. Can someone with a windows machine run this test without the annotation to see if this bug haven’t been fixed already. Thanks. Ismail ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Bug 22232] Conditional breakpoint test skipped for Windows
Thanks! I’ll remove these annotations from the test and submit a patch. Ismail > On Aug 20, 2019, at 2:11 PM, Stella Stamenova wrote: > > It looks like the tests are indeed passing on Windows. > > Thanks, > -Stella > > -Original Message- > From: lldb-dev On Behalf Of Ismail Bennani > via lldb-dev > Sent: Tuesday, August 20, 2019 11:50 AM > To: lldb-dev@lists.llvm.org > Subject: [lldb-dev] [Bug 22232] Conditional breakpoint test skipped for > Windows > > Hi, > > While looking at the LLDB test suite, I noticed that the test for conditional > breakpoints (TestBreakpointConditions.py) is skipped on Windows. > > The test has an annotation `skipIfWindows` with a comment: > `# Requires EE to support COFF on Windows > (https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fllvm.org%2Fpr22232&data=02%7C01%7CSTILIS%40microsoft.com%7C19421d8b4f9e4a84c1c308d7259f37e3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637019238092234576&sdata=niijsRJS%2ByKyoQ4nC8VI%2FyvGkDY6tcBVEas50142sFI%3D&reserved=0)` > > This bug was reported in 2015 and since then, I assume Windows gained support > for expression evaluation. Can someone with a windows machine run this test > without the annotation to see if this bug haven’t been fixed already. > > Thanks. > > Ismail > > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flldb-dev&data=02%7C01%7CSTILIS%40microsoft.com%7C19421d8b4f9e4a84c1c308d7259f37e3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637019238092234576&sdata=I86gaP%2Bj26JuxwDiP3Tf9h0YK3BlwfhUBXslE2vr7yk%3D&reserved=0 ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
