Re: [lldb-dev] RFC: full support for python files, and avoid using FILE* internally
> On Sep 24, 2019, at 10:04 AM, Ted Woodward wrote: > > A bit of a tangent, but I've been getting requests to debug Python and C++ > together. Things like TensorFlow start in Python, then call into C++ > libraries. Users want to be able to debug the Python code as Python (not > debugging into Python itself), then step into the C++ libraries. They want to > go up and down the stack, switching languages as needed. "What was my Python > code doing when it called into the library. Now what is the library doing?" Python does have a tool called faulthandler that can produce python stack traces when the interpreter experiences a segfault. I wonder if there’s a way for LLDB invoke that to get python backtraces in the debugger? ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: full support for python files, and avoid using FILE* internally
> On Sep 24, 2019, at 4:11 AM, Pavel Labath wrote: > > On 23/09/2019 20:54, Larry D'Anna wrote: >>> On Sep 23, 2019, at 7:11 AM, Pavel Labath wrote: >>> >>> On 20/09/2019 17:35, Larry D'Anna via lldb-dev wrote: Hi lldb-dev. I want to be able to use LLDB inside of iPython, so I can have mixed python and LLDB debug session. To this end, I’d like to update LLDB to have full support for python file objects, so the outputs of debugger commands can be redirected into iPython’s own streams. This however, is difficult to do, because LLDB makes use of FILE* streams in a number of places. This presents two problems. The first is that there is no really correct way to create SWIG typemaps that handle conversion to FILE* and get the ownership semantics correct. The second problem is that there is not a portable way to make a FILE* with arbitrary callbacks for reading and writing. On Darwin and BSD there’s funopen, and on linux there’s something else, and I don’t know if there’s any way on windows. I made an attempt at this a while ago using funopen a while ago, here: https://reviews.llvm.org/D38829 Zachary Turner suggested a more thorough approach. where instead of trying to use funopen to paper over all the use of FILE* streams, we should make lldb_private::File capable of doing the dynamic dispatch and excise all the unnecessary FILE* stuff in favor of lldb_private::File. That’s what I’ve done here: https://github.com/smoofra/llvm-project/tree/files I’ve posted the first few patches to phabricator for review. https://reviews.llvm.org/D67793 https://reviews.llvm.org/D67792 https://reviews.llvm.org/D67789 What do you think? >>> >>> >>> >>> Hello Larry, >>> >>> thanks for starting this thread. >>> >>> So, judging by your problem description, it sounds to me like you're >>> primarily interested in the SBCommandInterpreter::HandleCommand family of >>> functions (and by extension, the SBCommandReturnObject class). Would that >>> be a fair thing to say? >> Not really. I want to be able to embed a full LLDB session inside of >> iPython, which means redirecting anything that prints to the debugger's main >> output and error streams. Yes, in most cases that will be coming from >> HandleCommand(), but I really want to avoid the situation where some output >> that would normally be printed to the terminal is missed under iPython. > > Ok, that's fair. > >>> The reason I am asking this is that I'm wondering what is the scope of the >>> thing you're proposing to do (and then, whether this is the best way to >>> accomplish that). For instance, if we were only interested in the >>> HandleCommand api, then it might be possible to plug the python in at a >>> higher level (Stream instead of File). I am hoping that doing that might be >>> easier as the Stream class has a simpler interface, and already supports >>> multiple backing implementations (StreamFile, StreamString, ...). >>> >>> Also, doing that would allow to side step some complicated questions. One >>> of the reasons why getting rid of FILE* is so complicated (you're not the >>> first person to try that) is that there are some APIs (libedit mainly), >>> that we just cannot change, and which require a FILE*. >> I saw that. My strategy for dealing with that was to audit the codebase >> for any use of File::GetStream(). I found the only two places I could not >> remove the use of GetStream() was libedit and IOHandlerCursesGUI.In my >> prototype, I deal with that by checking for NULL from GetStream() before >> libedit or IOHandlerCursesGUI are enabled. In other words, If a File can >> produce a FILE*, it will. But you can still have a valid File that will >> return NULL from GetStream. If you set your debugger streams to Files >> that return NULL from GetStream, then libedit and the curses GUI will be >> disabled.I think this is a reasonable approach.For my use-case in >> particular, there is no need for either libedit or the curses gui, because >> the whole point is to use iPython as the gui. In general, libedit and >> curses only really make sense if the IO streams are a terminal anyway, so >> it’s not a problem to disable these features if the IO streams are >> redirected to python. > > Ok, that also sounds like a reasonable position to take. Might be the only > reasonable position, even. Theoretically, one might try to go the extra mile > and try to synthesize a FILE* using fopencookie et al. on platforms that > support that (the only platforms that support libedit and curses also happen > to have a fopencookie equivalent). That's probably overkill now, but it is > nice to have that option open for the future. > >>> If you do want to go with the more general change, then I'd like to ask you >>> to give a bit more detail about
Re: [lldb-dev] Hiding trampoline functions from the backtrace, is it possible ?
> On Sep 24, 2019, at 11:36 AM, Nat! via lldb-dev > wrote: > > > > On 24.09.19 19:28, Jim Ingham wrote: >> We've had many requests to elide some classes of entries in backtraces - >> like to mirror the Xcode display I mentioned previously. Most of these >> requests don't depend on the functions being marked artificial. So if we're >> going to do this, something more general than just "marked artificial" -> >> elided anyway. >> >> Jim >> >> > > Yes. > > Having done a little further research... Artificial won't work for general > cases anyway, since it's restricted to inline code (for some reason) on gcc > and clang. I wonder why, since for a function the only real effect is to emit > DW_AT_artificial (AFAIK). The restriction seems arbitrary and DWARF wouldn't > mind.. But the compilers do, so it seems out anyway. > > DW_AT_trampoline isn't supported by llvm. As I read the description of > DW_AT_trampoline, its more like a hardcoded vector (a->b), so not useful for > cases like objc_msgSend, where you don't know the destination a priori. > > If I look at the DWARF spec, I don't see any other way to mark a function as > "boring". I still think this would be a good thing, as this would be useful > for other debuggers as well, which could instantly work. Also a lot of code > in the lldb Trampoline classes, for step-in and step-out could probably just > be removed. I don't think that is right for "step-in". As you said above, in the classic example of a trampoline: objc_msgSend you can't statically know the destination. So the DWARF can't help resolve this; you would still need to do the work the lldb trampoline classes do at runtime. step-out past trampolines could just "keep stepping past boring functions". There's no need to support this for ObjC - at least the Apple & NeXT versions - since the dispatch function is a tail call function. But we do do something like for Swift. But that part is very little code compared to figuring how to step in correctly. Jim > > Ciao >Nat! > > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Hiding trampoline functions from the backtrace, is it possible ?
On 24.09.19 19:28, Jim Ingham wrote: We've had many requests to elide some classes of entries in backtraces - like to mirror the Xcode display I mentioned previously. Most of these requests don't depend on the functions being marked artificial. So if we're going to do this, something more general than just "marked artificial" -> elided anyway. Jim Yes. Having done a little further research... Artificial won't work for general cases anyway, since it's restricted to inline code (for some reason) on gcc and clang. I wonder why, since for a function the only real effect is to emit DW_AT_artificial (AFAIK). The restriction seems arbitrary and DWARF wouldn't mind.. But the compilers do, so it seems out anyway. DW_AT_trampoline isn't supported by llvm. As I read the description of DW_AT_trampoline, its more like a hardcoded vector (a->b), so not useful for cases like objc_msgSend, where you don't know the destination a priori. If I look at the DWARF spec, I don't see any other way to mark a function as "boring". I still think this would be a good thing, as this would be useful for other debuggers as well, which could instantly work. Also a lot of code in the lldb Trampoline classes, for step-in and step-out could probably just be removed. Ciao Nat! ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Hiding trampoline functions from the backtrace, is it possible ?
We've had many requests to elide some classes of entries in backtraces - like to mirror the Xcode display I mentioned previously. Most of these requests don't depend on the functions being marked artificial. So if we're going to do this, something more general than just "marked artificial" -> elided anyway. Jim > On Sep 24, 2019, at 3:01 AM, Nat! via lldb-dev > wrote: > > Nat! schrieb am 24.09.19 um 00:28: >> On 23.09.19 19:22, Adrian Prantl wrote: >>> I think the best mechanism for this would be to ensure that the trampolines >>> are marked up as DW_AT_artificial and/or DW_AT_trampoline by the compiler. >>> I'm pretty sure LLDB then already knows how to hide artificial frames >>> (somebody else can probably provide pointers for how that works). >>> >>> -- adrian >>> >> That's a good idea. I can just put __attribute__((artificial)__ on my >> dispatch functions. That's the low-hanging-fruit code I like :) >> And though it might not fully work yet with lldb , it may in the future. > > Unfortunately clang complains that the "'artificial' attribute only applies > to inline functions" (why ?). Bummer. > > Ciao > Nat! > > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: full support for python files, and avoid using FILE* internally
A bit of a tangent, but I've been getting requests to debug Python and C++ together. Things like TensorFlow start in Python, then call into C++ libraries. Users want to be able to debug the Python code as Python (not debugging into Python itself), then step into the C++ libraries. They want to go up and down the stack, switching languages as needed. "What was my Python code doing when it called into the library. Now what is the library doing?" > -Original Message- > From: lldb-dev On Behalf Of Pavel Labath > via lldb-dev > Sent: Tuesday, September 24, 2019 6:11 AM > To: Larry D'Anna > Cc: lldb-dev@lists.llvm.org > Subject: [EXT] Re: [lldb-dev] RFC: full support for python files, and avoid > using > FILE* internally > > On 23/09/2019 20:54, Larry D'Anna wrote: > > > > > >> On Sep 23, 2019, at 7:11 AM, Pavel Labath wrote: > >> > >> On 20/09/2019 17:35, Larry D'Anna via lldb-dev wrote: > >>> Hi lldb-dev. > >>> I want to be able to use LLDB inside of iPython, so I can have mixed > python and LLDB debug session. > >>> To this end, I’d like to update LLDB to have full support for python > >>> file objects, so the outputs of > debugger commands can be redirected into iPython’s own streams. > >>> This however, is difficult to do, because LLDB makes use of FILE* streams > in a number of places. This presents two problems. The first is that there > is > no really > >>> correct way to create SWIG typemaps that handle conversion to FILE* > and get the ownership semantics correct. The second problem is that there > is not a portable > >>> way to make a FILE* with arbitrary callbacks for reading and writing. On > Darwin and BSD there’s funopen, and on linux there’s something else, and I > don’t know if > >>> there’s any way on windows. > >>> I made an attempt at this a while ago using funopen a while ago, here: > >>> https://reviews.llvm.org/D38829 > >>> Zachary Turner suggested a more thorough approach. where instead of > >>> trying to use funopen to paper over all the use of FILE* streams, we > should make lldb_private::File capable of doing the dynamic dispatch and > excise all the unnecessary FILE* stuff in favor of lldb_private::File. > >>> That’s what I’ve done here: > >>> https://github.com/smoofra/llvm-project/tree/files > >>> I’ve posted the first few patches to phabricator for review. > >>> https://reviews.llvm.org/D67793 > >>> https://reviews.llvm.org/D67792 > >>> https://reviews.llvm.org/D67789 > >>> What do you think? > >> > >> > >> > >> Hello Larry, > >> > >> thanks for starting this thread. > >> > >> So, judging by your problem description, it sounds to me like you're > primarily interested in the SBCommandInterpreter::HandleCommand family of > functions (and by extension, the SBCommandReturnObject class). Would that > be a fair thing to say? > > > > Not really. I want to be able to embed a full LLDB session inside of > > iPython, > which means redirecting anything that prints to the debugger's main output > and error streams. Yes, in most cases that will be coming from > HandleCommand(), but I really want to avoid the situation where some output > that would normally be printed to the terminal is missed under iPython. > > Ok, that's fair. > > > > >> The reason I am asking this is that I'm wondering what is the scope of the > thing you're proposing to do (and then, whether this is the best way to > accomplish that). For instance, if we were only interested in the > HandleCommand api, then it might be possible to plug the python in at a > higher level (Stream instead of File). I am hoping that doing that might be > easier as the Stream class has a simpler interface, and already supports > multiple backing implementations (StreamFile, StreamString, ...). > >> > >> Also, doing that would allow to side step some complicated questions. One > of the reasons why getting rid of FILE* is so complicated (you're not the > first > person to try that) is that there are some APIs (libedit mainly), that we just > cannot change, and which require a FILE*. > > > > I saw that. My strategy for dealing with that was to audit the codebase > > for > any use of File::GetStream(). I found the only two places I could not remove > the use of GetStream() was libedit and IOHandlerCursesGUI.In my > prototype, I deal with that by checking for NULL from GetStream() before > libedit or IOHandlerCursesGUI are enabled. In other words, If a File can > produce a FILE*, it will. But you can still have a valid File that will > return NULL > from GetStream. If you set your debugger streams to Files that return > NULL > from GetStream, then libedit and the curses GUI will be disabled.I think > this > is a reasonable approach.For my use-case in particular, there is no need > for > either libedit or the curses gui, because the whole point is to use iPython as > the gui. In general, libedit and curses only really make sense if the IO > streams are a terminal anyway, so it’s no
Re: [lldb-dev] RFC: full support for python files, and avoid using FILE* internally
On 23/09/2019 20:54, Larry D'Anna wrote: On Sep 23, 2019, at 7:11 AM, Pavel Labath wrote: On 20/09/2019 17:35, Larry D'Anna via lldb-dev wrote: Hi lldb-dev. I want to be able to use LLDB inside of iPython, so I can have mixed python and LLDB debug session. To this end, I’d like to update LLDB to have full support for python file objects, so the outputs of debugger commands can be redirected into iPython’s own streams. This however, is difficult to do, because LLDB makes use of FILE* streams in a number of places. This presents two problems. The first is that there is no really correct way to create SWIG typemaps that handle conversion to FILE* and get the ownership semantics correct. The second problem is that there is not a portable way to make a FILE* with arbitrary callbacks for reading and writing. On Darwin and BSD there’s funopen, and on linux there’s something else, and I don’t know if there’s any way on windows. I made an attempt at this a while ago using funopen a while ago, here: https://reviews.llvm.org/D38829 Zachary Turner suggested a more thorough approach. where instead of trying to use funopen to paper over all the use of FILE* streams, we should make lldb_private::File capable of doing the dynamic dispatch and excise all the unnecessary FILE* stuff in favor of lldb_private::File. That’s what I’ve done here: https://github.com/smoofra/llvm-project/tree/files I’ve posted the first few patches to phabricator for review. https://reviews.llvm.org/D67793 https://reviews.llvm.org/D67792 https://reviews.llvm.org/D67789 What do you think? Hello Larry, thanks for starting this thread. So, judging by your problem description, it sounds to me like you're primarily interested in the SBCommandInterpreter::HandleCommand family of functions (and by extension, the SBCommandReturnObject class). Would that be a fair thing to say? Not really. I want to be able to embed a full LLDB session inside of iPython, which means redirecting anything that prints to the debugger's main output and error streams. Yes, in most cases that will be coming from HandleCommand(), but I really want to avoid the situation where some output that would normally be printed to the terminal is missed under iPython. Ok, that's fair. The reason I am asking this is that I'm wondering what is the scope of the thing you're proposing to do (and then, whether this is the best way to accomplish that). For instance, if we were only interested in the HandleCommand api, then it might be possible to plug the python in at a higher level (Stream instead of File). I am hoping that doing that might be easier as the Stream class has a simpler interface, and already supports multiple backing implementations (StreamFile, StreamString, ...). Also, doing that would allow to side step some complicated questions. One of the reasons why getting rid of FILE* is so complicated (you're not the first person to try that) is that there are some APIs (libedit mainly), that we just cannot change, and which require a FILE*. I saw that. My strategy for dealing with that was to audit the codebase for any use of File::GetStream(). I found the only two places I could not remove the use of GetStream() was libedit and IOHandlerCursesGUI.In my prototype, I deal with that by checking for NULL from GetStream() before libedit or IOHandlerCursesGUI are enabled. In other words, If a File can produce a FILE*, it will. But you can still have a valid File that will return NULL from GetStream. If you set your debugger streams to Files that return NULL from GetStream, then libedit and the curses GUI will be disabled.I think this is a reasonable approach.For my use-case in particular, there is no need for either libedit or the curses gui, because the whole point is to use iPython as the gui. In general, libedit and curses only really make sense if the IO streams are a terminal anyway, so it’s not a problem to disable these features if the IO streams are redirected to python. Ok, that also sounds like a reasonable position to take. Might be the only reasonable position, even. Theoretically, one might try to go the extra mile and try to synthesize a FILE* using fopencookie et al. on platforms that support that (the only platforms that support libedit and curses also happen to have a fopencookie equivalent). That's probably overkill now, but it is nice to have that option open for the future. If you do want to go with the more general change, then I'd like to ask you to give a bit more detail about the your vision of the new role of the lldb_private::File class and its interaction with other major lldb components (SBFile, StreamFile, ???). My understanding (it's been a while since I looked at this in detail) is that the File class can be constructed from both FILE* and a file descriptor and (crucially) it is also able to give back these underlying objects, includ
Re: [lldb-dev] Hiding trampoline functions from the backtrace, is it possible ?
Nat! schrieb am 24.09.19 um 00:28: On 23.09.19 19:22, Adrian Prantl wrote: I think the best mechanism for this would be to ensure that the trampolines are marked up as DW_AT_artificial and/or DW_AT_trampoline by the compiler. I'm pretty sure LLDB then already knows how to hide artificial frames (somebody else can probably provide pointers for how that works). -- adrian That's a good idea. I can just put __attribute__((artificial)__ on my dispatch functions. That's the low-hanging-fruit code I like :) And though it might not fully work yet with lldb , it may in the future. Unfortunately clang complains that the "'artificial' attribute only applies to inline functions" (why ?). Bummer. Ciao Nat! ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Release-testers] [9.0.0 Release] Release Candidate 6 is here
Hi, Uploaded f8f3e6bdd640079a140a7ada4eb6f5f05aeae125cf54b94d44f733b0e8691dc2 clang+llvm-9.0.0-aarch64-linux-gnu.tar.xz ff6046bf98dbc85d7cb0c3c70456bc002b99a809bfc115657db2683ba61752ec clang+llvm-9.0.0-armv7a-linux-gnueabihf.tar.xz Cheers, Diana On Thu, 19 Sep 2019 at 16:56, Hans Wennborg via Release-testers wrote: > > On Tue, Sep 17, 2019 at 4:05 PM Hans Wennborg wrote: > > > > Hello everyone, > > > > 9.0.0-rc6 was just tagged from the release_90 branch at r372100. In > > the Git monorepo, it's tagged as llvmorg-9.0.0-rc6. > > This has now been tagged as the final 9.0.0 release. In the Git > monorepo, it's tagged as llvmorg-9.0.0. > > The official release announcement will follow as soon as the source > tarballs and docs are ready on the web page. > > Testers: please start building the final binaries, and I'll add them > as they become ready. > > Many thanks for your hard work! > Hans > ___ > Release-testers mailing list > release-test...@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/release-testers ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
