Re: [HACKERS] Printing bitmap objects in the debugger
Thanks a lot. On Fri, Sep 16, 2016 at 7:07 PM, Tom Lane wrote: > Ashutosh Bapat writes: >>> I'd suggest that this is parallel to nodeToString() and therefore >>> (a) should be placed beside it, > >> Done. Added it after nodeToString(). > > Pushed, thanks. > > regards, tom lane -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Ashutosh Bapat writes: >> I'd suggest that this is parallel to nodeToString() and therefore >> (a) should be placed beside it, > Done. Added it after nodeToString(). Pushed, thanks. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
> > I'd suggest that this is parallel to nodeToString() and therefore > (a) should be placed beside it, Done. Added it after nodeToString(). > (b) should be named like it, > bmsToString() perhaps, bmsToString() is fine. Used that name. > and (c) should look more like it internally. > Done. I have also added a declaration for this function in nodes.h after definition of struct Bitmapset. WIthout this declaration compiler gives warning "no previous declaration" of this function. Tested it under the debugger Breakpoint 1, make_join_rel (root=0x20cafb0, rel1=0x20e2998, rel2=0x20dd2c0) at joinrels.c:664 (gdb) p bmsToString(rel1->relids) $1 = 0x2102fd0 "(b 1 3)" (gdb) p bmsToString(rel2->relids) $2 = 0x2104bc0 "(b 4)" (gdb) p bmsToString(joinrelids) $3 = 0x2104fd8 "(b 1 3 4)" (gdb) p bmsToString(joinrel->relids) $4 = 0x2105998 "(b 1 3 4)" (gdb) p bmsToString(joinrel->lateral_relids) $5 = 0x2105db0 "(b)" (gdb) p joinrel->lateral_relids $6 = (Relids) 0x0 -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 29b7712..e1bbcf7 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -3886,10 +3886,25 @@ outNode(StringInfo str, const void *obj) char * nodeToString(const void *obj) { StringInfoData str; /* see stringinfo.h for an explanation of this maneuver */ initStringInfo(&str); outNode(&str, obj); return str.data; } + +/* + * bmsToNode - + * returns the ascii representation of the Bitmapset as a palloc'd string + */ +char * +bmsToString(const Bitmapset *bms) +{ + StringInfoData str; + + /* see stringinfo.h for an explanation of this maneuver */ + initStringInfo(&str); + outBitmapset(&str, bms); + return str.data; +} diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 2f7efa8..b239b99 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -554,20 +554,21 @@ extern PGDLLIMPORT Node *newNodeMacroHolder; extern char *nodeToString(const void *obj); struct Bitmapset;/* not to include bitmapset.h here */ struct StringInfoData; /* not to include stringinfo.h here */ extern void outNode(struct StringInfoData *str, const void *obj); extern void outToken(struct StringInfoData *str, const char *s); extern void outBitmapset(struct StringInfoData *str, const struct Bitmapset *bms); extern void outDatum(struct StringInfoData *str, uintptr_t value, int typlen, bool typbyval); +extern char *bmsToString(const struct Bitmapset *bms); /* * nodes/{readfuncs.c,read.c} */ extern void *stringToNode(char *str); extern struct Bitmapset *readBitmapset(void); extern uintptr_t readDatum(bool typbyval); extern bool *readBoolCols(int numCols); extern int *readIntCols(int numCols); extern Oid *readOidCols(int numCols); -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Robert Haas writes: > This seems like a very complicated mechanism of substituting for a > very simple patch. Well, if we're gonna do it, then let's just do it, but please let's have a patch that doesn't look like somebody's temporary debugging kluge. I'd suggest that this is parallel to nodeToString() and therefore (a) should be placed beside it, (b) should be named like it, bmsToString() perhaps, and (c) should look more like it internally. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Thu, Sep 15, 2016 at 7:55 PM, Pavan Deolasee wrote: > > (lldb) script print print_bms_members(lldb.frame.FindVariable ("a")) > nwords = 1 bitmap: 0x200 > > Or even this if lldb.frame.FindVariable() is pushed inside the function: (lldb) script print print_bms_members('a') nwords = 1 bitmap: 0x200 Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] Printing bitmap objects in the debugger
On Thu, Sep 15, 2016 at 7:38 PM, Robert Haas wrote: > > > This seems like a very complicated mechanism of substituting for a > very simple patch. I don't have objection to the patch per se. The point of posting this was just to share other mechanisms that exists. BTW advantage of using debugger scripts is that they also work while inspecting core dumps. > Your LLDB script is about the same number of lines > as Ashutosh's patch and only works for people who use LLDB. Alvaro pointed out that gdb also have similar capabilities. > Plus, > once you write it, you've got to enter the Python interpreter to use > it and then run three more lines of code that aren't easy to remember. > In contrast, with Ashutosh's proposed patch, you just write: > > p bms_to_char(bms_object) > > I learnt this yesterday and I am sure there are easier ways to do the same thing. I just don't know. For example, you can also do this: (lldb) script print print_bms_members(lldb.frame.FindVariable ("a")) nwords = 1 bitmap: 0x200 It's still slightly cumbersome, but better than entering the interpreter. > ...and you're done. Now, I grant that his approach bloats the binary > and yours does not, but nobody complains about pprint() bloating the > binary. Sure. I wasn't aware of existence of pprint() either and may be that's enough from debugging perspective. When I tried that yesterday, the output went to the logfile instead of coming on the debugger prompt. May be I did something wrong or may be that's not inconvenient for those who use it regularly. So yeah, no objections to the patch. I was happy to discover what I did and thought of sharing assuming others might find it useful too. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 8:01 AM, Pavan Deolasee wrote: > On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee > wrote: >> lately I'm using LVM debugger (which probably does not have something >> equivalent), > > > And I was so clueless about lldb's powerful scripting interface. For > example, you can write something like this in bms_utils.py: > > import lldb > > def print_bms_members (bms): > words = bms.GetChildMemberWithName("words") > nwords = int(bms.GetChildMemberWithName("nwords").GetValue()) > > ret = 'nwords = {0} bitmap: '.format(nwords,) > for i in range(0, nwords): > ret += hex(int(words.GetChildAtIndex(0, lldb.eNoDynamicValues, > True).GetValue())) > > return ret > > And then do this while attached to lldb debugger: > > Process 99659 stopped > * thread #1: tid = 0x59ba69, 0x0001090b012f > postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673, > queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 > frame #0: 0x0001090b012f > postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673 >670 int wordnum, >671 bitnum; >672 > -> 673 if (x < 0) >674 elog(ERROR, "negative bitmapset member not allowed"); >675 if (a == NULL) >676 return bms_make_singleton(x); > (lldb) script > Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D. from bms_utils import * bms = lldb.frame.FindVariable ("a") print print_bms_members(bms) > nwords = 1 bitmap: 0x200 > > > The complete API reference is available here > http://lldb.llvm.org/python_reference/index.html > > Looks like an interesting SoC project to write useful lldb/gdb scripts to > print internal structures for ease of debugging :-) This seems like a very complicated mechanism of substituting for a very simple patch. Your LLDB script is about the same number of lines as Ashutosh's patch and only works for people who use LLDB. Plus, once you write it, you've got to enter the Python interpreter to use it and then run three more lines of code that aren't easy to remember. In contrast, with Ashutosh's proposed patch, you just write: p bms_to_char(bms_object) ...and you're done. Now, I grant that his approach bloats the binary and yours does not, but nobody complains about pprint() bloating the binary. If that's actually an issue people are really concerned about then let's just reject this and Ashutosh can patch his local copy. If that's not a serious problem then let's take the patch. If anything, I think this discussion shows that LLDB macros are probably too much of a pain to be seriously considered for everyday use, unless perhaps you're the sort of person who plans to spend the day inside the Python shell anyway. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 8:45 PM, Alvaro Herrera wrote: > Tom Lane wrote: >> Ashutosh Bapat writes: >> > While working on partition-wise join, I had to examine Relids objects >> > many times. Printing the Bitmapset::words[] in binary format and then >> > inferring the relids takes time and is error prone. >> >> FWIW, I generally rely on pprint() to look at planner data structures >> from the debugger. > > Also: > http://blog.pgaddict.com/posts/making-debugging-with-gdb-a-bit-easier > This may not address the issue directly, but it's probably very helpful. Thanks for the reference. I think this is something similar to what Pavan suggested in the mail thread. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
>> Alvaro Herrera writes: >>> I don't understand. Why don't you just use "call pprint(the bitmapset)" >>> in the debugger? >> >> Bitmapsets aren't Nodes, so pprint doesn't work directly on them. >> I usually find that I can pprint some node containing the value(s) >> I'm interested in, but maybe that isn't working for Ashutosh's >> particular case. that's right. > > There are many loose (ie, not inside any Node) Relids variables within the > optimizer code. Perhaps Ashutosh ended up needing to look at those a lot. that's right too. In joinrels.c for example we are manipulating Relids so many times. [ashutosh@ubuntu pg_head]grep bms_ src/backend/optimizer/path/joinrels.c | wc -l 69 There are many other instances of this in other optimizer and planner files. There are other places where we manipulate Bitmapsets. And not every Relids object computed is contained in a Node. So, pprint() doesn't help much. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Thu, Sep 15, 2016 at 2:58 PM, Ashutosh Bapat wrote: > On Wed, Sep 14, 2016 at 5:31 PM, Pavan Deolasee > wrote: >> The complete API reference is available here >> http://lldb.llvm.org/python_reference/index.html >> >> Looks like an interesting SoC project to write useful lldb/gdb scripts to >> print internal structures for ease of debugging :-) >> > > +1, if we can include something like that in the repository so as to > avoid every developer maintaining a script of his/her own. +1. Even if one finishes by doing manual modifications of some of them, it is always good to have a central point of reference. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 5:31 PM, Pavan Deolasee wrote: > > On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee > wrote: >> >> >> >> lately I'm using LVM debugger (which probably does not have something >> equivalent), > > > And I was so clueless about lldb's powerful scripting interface. For > example, you can write something like this in bms_utils.py: > > import lldb > > def print_bms_members (bms): > words = bms.GetChildMemberWithName("words") > nwords = int(bms.GetChildMemberWithName("nwords").GetValue()) > > ret = 'nwords = {0} bitmap: '.format(nwords,) > for i in range(0, nwords): > ret += hex(int(words.GetChildAtIndex(0, lldb.eNoDynamicValues, > True).GetValue())) > > return ret > Thanks a lot for digging into it. > And then do this while attached to lldb debugger: > > Process 99659 stopped > * thread #1: tid = 0x59ba69, 0x0001090b012f > postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673, > queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 > frame #0: 0x0001090b012f > postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673 >670 int wordnum, >671 bitnum; >672 > -> 673 if (x < 0) >674 elog(ERROR, "negative bitmapset member not allowed"); >675 if (a == NULL) >676 return bms_make_singleton(x); > (lldb) script > Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D. from bms_utils import * bms = lldb.frame.FindVariable ("a") print print_bms_members(bms) > nwords = 1 bitmap: 0x200 > I can get that kind of output by command p *bms p/x (or p/b) *bms->words@(bms->nwords) in gdb. But I can certainly extend the script you wrote above to print more meaningful output similar to outBitmapset(). But then this would be specific to LLVM. GDB too seems to have a similar interface https://sourceware.org/gdb/wiki/PythonGdbTutorial, so I can probably use above script with some modifications with GDB as well. Python script will be easier to maintain as compared to maintaining a patch that needs to be applied and compiled. Said that, I am not sure if every debugger supported on every platform we support has these features. Or may be developers work on only those platforms which have such powerful debuggers, so it's ok. Every debugger usually has much easier way to call a function and print its output, so having a function like the one I have in the patch makes things easy for all the debuggers and may be developers not familiar with python. > > The complete API reference is available here > http://lldb.llvm.org/python_reference/index.html > > Looks like an interesting SoC project to write useful lldb/gdb scripts to > print internal structures for ease of debugging :-) > +1, if we can include something like that in the repository so as to avoid every developer maintaining a script of his/her own. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On 2016/09/15 0:04, Tom Lane wrote: > Alvaro Herrera writes: >> I don't understand. Why don't you just use "call pprint(the bitmapset)" >> in the debugger? > > Bitmapsets aren't Nodes, so pprint doesn't work directly on them. > I usually find that I can pprint some node containing the value(s) > I'm interested in, but maybe that isn't working for Ashutosh's > particular case. There are many loose (ie, not inside any Node) Relids variables within the optimizer code. Perhaps Ashutosh ended up needing to look at those a lot. Thanks, Amit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Tom Lane wrote: > Ashutosh Bapat writes: > > While working on partition-wise join, I had to examine Relids objects > > many times. Printing the Bitmapset::words[] in binary format and then > > inferring the relids takes time and is error prone. > > FWIW, I generally rely on pprint() to look at planner data structures > from the debugger. Also: http://blog.pgaddict.com/posts/making-debugging-with-gdb-a-bit-easier This may not address the issue directly, but it's probably very helpful. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Ashutosh Bapat writes: > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. FWIW, I generally rely on pprint() to look at planner data structures from the debugger. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Alvaro Herrera writes: > I don't understand. Why don't you just use "call pprint(the bitmapset)" > in the debugger? Bitmapsets aren't Nodes, so pprint doesn't work directly on them. I usually find that I can pprint some node containing the value(s) I'm interested in, but maybe that isn't working for Ashutosh's particular case. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Ashutosh Bapat wrote: > Hi All, > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. Instead I wrote a > function bms_to_char() which allocates a StringInfo, calls > outBitmapset() to decode Bitmapset as a set of integers and returns > the string. In order to examine a Relids object all one has to do is > execute 'p bms_to_char(bms_object) under gdb. I don't understand. Why don't you just use "call pprint(the bitmapset)" in the debugger? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee wrote: > > > lately I'm using LVM debugger (which probably does not have something > equivalent), > And I was so clueless about lldb's powerful scripting interface. For example, you can write something like this in bms_utils.py: import lldb def print_bms_members (bms): words = bms.GetChildMemberWithName("words") nwords = int(bms.GetChildMemberWithName("nwords").GetValue()) ret = 'nwords = {0} bitmap: '.format(nwords,) for i in range(0, nwords): ret += hex(int(words.GetChildAtIndex(0, lldb.eNoDynamicValues, True).GetValue())) return ret And then do this while attached to lldb debugger: Process 99659 stopped * thread #1: tid = 0x59ba69, 0x0001090b012f postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 frame #0: 0x0001090b012f postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673 670 int wordnum, 671 bitnum; 672 -> 673 if (x < 0) 674 elog(ERROR, "negative bitmapset member not allowed"); 675 if (a == NULL) 676 return bms_make_singleton(x); (lldb) script Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D. >>> from bms_utils import * >>> bms = lldb.frame.FindVariable ("a") >>> print print_bms_members(bms) nwords = 1 bitmap: 0x200 The complete API reference is available here http://lldb.llvm.org/python_reference/index.html Looks like an interesting SoC project to write useful lldb/gdb scripts to print internal structures for ease of debugging :-) Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 3:43 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > Hi All, > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. Instead I wrote a > function bms_to_char() which allocates a StringInfo, calls > outBitmapset() to decode Bitmapset as a set of integers and returns > the string. In order to examine a Relids object all one has to do is > execute 'p bms_to_char(bms_object) under gdb. > > Can we not do this as gdb macros? My knowledge is rusty in this area and lately I'm using LVM debugger (which probably does not have something equivalent), but I believe gdb allows you to write useful macros. As a bonus, you can then use them even for inspecting core files. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
[HACKERS] Printing bitmap objects in the debugger
Hi All, While working on partition-wise join, I had to examine Relids objects many times. Printing the Bitmapset::words[] in binary format and then inferring the relids takes time and is error prone. Instead I wrote a function bms_to_char() which allocates a StringInfo, calls outBitmapset() to decode Bitmapset as a set of integers and returns the string. In order to examine a Relids object all one has to do is execute 'p bms_to_char(bms_object) under gdb. Is there a way, this can be included in the code? If it's available in the code, developers don't have to apply the patch and compile it for debugging. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 29b7712..b4cae11 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -93,20 +93,22 @@ outNode(str, node->fldname)) /* Write a bitmapset field */ #define WRITE_BITMAPSET_FIELD(fldname) \ (appendStringInfo(str, " :" CppAsString(fldname) " "), \ _outBitmapset(str, node->fldname)) #define booltostr(x) ((x) ? "true" : "false") +char *bms_to_char(const Bitmapset *bms); + /* * _outToken * Convert an ordinary string (eg, an identifier) into a form that * will be decoded back to a plain token by read.c's functions. * * If a null or empty string is given, it is encoded as "<>". */ static void _outToken(StringInfo str, const char *s) @@ -204,20 +206,31 @@ _outBitmapset(StringInfo str, const Bitmapset *bms) } /* for use by extensions which define extensible nodes */ void outBitmapset(StringInfo str, const Bitmapset *bms) { _outBitmapset(str, bms); } /* + * TODO: remove, used for debugging through gdb. + */ +char * +bms_to_char(const Bitmapset *bms) +{ + StringInfo str = makeStringInfo(); + outBitmapset(str, bms); + return str->data; +} + +/* * Print the value of a Datum given its type. */ void outDatum(StringInfo str, Datum value, int typlen, bool typbyval) { Size length, i; char *s; length = datumGetSize(value, typbyval, typlen); -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers