Re: [HACKERS] Printing bitmap objects in the debugger
Thanks a lot. On Fri, Sep 16, 2016 at 7:07 PM, Tom Lane wrote: > Ashutosh Bapat writes: >>> I'd suggest that this is parallel to nodeToString() and therefore >>> (a) should be placed beside it, > >> Done. Added it after nodeToString(). > > Pushed, thanks. > > regards, tom lane -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Ashutosh Bapat writes: >> I'd suggest that this is parallel to nodeToString() and therefore >> (a) should be placed beside it, > Done. Added it after nodeToString(). Pushed, thanks. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
>
> I'd suggest that this is parallel to nodeToString() and therefore
> (a) should be placed beside it,
Done. Added it after nodeToString().
> (b) should be named like it,
> bmsToString() perhaps,
bmsToString() is fine. Used that name.
> and (c) should look more like it internally.
>
Done.
I have also added a declaration for this function in nodes.h after
definition of struct Bitmapset. WIthout this declaration compiler
gives warning "no previous declaration" of this function.
Tested it under the debugger
Breakpoint 1, make_join_rel (root=0x20cafb0, rel1=0x20e2998,
rel2=0x20dd2c0) at joinrels.c:664
(gdb) p bmsToString(rel1->relids)
$1 = 0x2102fd0 "(b 1 3)"
(gdb) p bmsToString(rel2->relids)
$2 = 0x2104bc0 "(b 4)"
(gdb) p bmsToString(joinrelids)
$3 = 0x2104fd8 "(b 1 3 4)"
(gdb) p bmsToString(joinrel->relids)
$4 = 0x2105998 "(b 1 3 4)"
(gdb) p bmsToString(joinrel->lateral_relids)
$5 = 0x2105db0 "(b)"
(gdb) p joinrel->lateral_relids
$6 = (Relids) 0x0
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 29b7712..e1bbcf7 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3886,10 +3886,25 @@ outNode(StringInfo str, const void *obj)
char *
nodeToString(const void *obj)
{
StringInfoData str;
/* see stringinfo.h for an explanation of this maneuver */
initStringInfo(&str);
outNode(&str, obj);
return str.data;
}
+
+/*
+ * bmsToNode -
+ * returns the ascii representation of the Bitmapset as a palloc'd string
+ */
+char *
+bmsToString(const Bitmapset *bms)
+{
+ StringInfoData str;
+
+ /* see stringinfo.h for an explanation of this maneuver */
+ initStringInfo(&str);
+ outBitmapset(&str, bms);
+ return str.data;
+}
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 2f7efa8..b239b99 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -554,20 +554,21 @@ extern PGDLLIMPORT Node *newNodeMacroHolder;
extern char *nodeToString(const void *obj);
struct Bitmapset;/* not to include bitmapset.h here */
struct StringInfoData; /* not to include stringinfo.h here */
extern void outNode(struct StringInfoData *str, const void *obj);
extern void outToken(struct StringInfoData *str, const char *s);
extern void outBitmapset(struct StringInfoData *str,
const struct Bitmapset *bms);
extern void outDatum(struct StringInfoData *str, uintptr_t value,
int typlen, bool typbyval);
+extern char *bmsToString(const struct Bitmapset *bms);
/*
* nodes/{readfuncs.c,read.c}
*/
extern void *stringToNode(char *str);
extern struct Bitmapset *readBitmapset(void);
extern uintptr_t readDatum(bool typbyval);
extern bool *readBoolCols(int numCols);
extern int *readIntCols(int numCols);
extern Oid *readOidCols(int numCols);
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Robert Haas writes: > This seems like a very complicated mechanism of substituting for a > very simple patch. Well, if we're gonna do it, then let's just do it, but please let's have a patch that doesn't look like somebody's temporary debugging kluge. I'd suggest that this is parallel to nodeToString() and therefore (a) should be placed beside it, (b) should be named like it, bmsToString() perhaps, and (c) should look more like it internally. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Thu, Sep 15, 2016 at 7:55 PM, Pavan Deolasee
wrote:
>
> (lldb) script print print_bms_members(lldb.frame.FindVariable ("a"))
> nwords = 1 bitmap: 0x200
>
>
Or even this if lldb.frame.FindVariable() is pushed inside the function:
(lldb) script print print_bms_members('a')
nwords = 1 bitmap: 0x200
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] Printing bitmap objects in the debugger
On Thu, Sep 15, 2016 at 7:38 PM, Robert Haas wrote:
>
>
> This seems like a very complicated mechanism of substituting for a
> very simple patch.
I don't have objection to the patch per se. The point of posting this was
just to share other mechanisms that exists. BTW advantage of using debugger
scripts is that they also work while inspecting core dumps.
> Your LLDB script is about the same number of lines
> as Ashutosh's patch and only works for people who use LLDB.
Alvaro pointed out that gdb also have similar capabilities.
> Plus,
> once you write it, you've got to enter the Python interpreter to use
> it and then run three more lines of code that aren't easy to remember.
>
In contrast, with Ashutosh's proposed patch, you just write:
>
> p bms_to_char(bms_object)
>
>
I learnt this yesterday and I am sure there are easier ways to do the same
thing. I just don't know. For example, you can also do this:
(lldb) script print print_bms_members(lldb.frame.FindVariable ("a"))
nwords = 1 bitmap: 0x200
It's still slightly cumbersome, but better than entering the interpreter.
> ...and you're done. Now, I grant that his approach bloats the binary
> and yours does not, but nobody complains about pprint() bloating the
> binary.
Sure. I wasn't aware of existence of pprint() either and may be that's
enough from debugging perspective. When I tried that yesterday, the output
went to the logfile instead of coming on the debugger prompt. May be I did
something wrong or may be that's not inconvenient for those who use it
regularly.
So yeah, no objections to the patch. I was happy to discover what I did and
thought of sharing assuming others might find it useful too.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 8:01 AM, Pavan Deolasee
wrote:
> On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee
> wrote:
>> lately I'm using LVM debugger (which probably does not have something
>> equivalent),
>
>
> And I was so clueless about lldb's powerful scripting interface. For
> example, you can write something like this in bms_utils.py:
>
> import lldb
>
> def print_bms_members (bms):
> words = bms.GetChildMemberWithName("words")
> nwords = int(bms.GetChildMemberWithName("nwords").GetValue())
>
> ret = 'nwords = {0} bitmap: '.format(nwords,)
> for i in range(0, nwords):
> ret += hex(int(words.GetChildAtIndex(0, lldb.eNoDynamicValues,
> True).GetValue()))
>
> return ret
>
> And then do this while attached to lldb debugger:
>
> Process 99659 stopped
> * thread #1: tid = 0x59ba69, 0x0001090b012f
> postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673,
> queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
> frame #0: 0x0001090b012f
> postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673
>670 int wordnum,
>671 bitnum;
>672
> -> 673 if (x < 0)
>674 elog(ERROR, "negative bitmapset member not allowed");
>675 if (a == NULL)
>676 return bms_make_singleton(x);
> (lldb) script
> Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
from bms_utils import *
bms = lldb.frame.FindVariable ("a")
print print_bms_members(bms)
> nwords = 1 bitmap: 0x200
>
>
> The complete API reference is available here
> http://lldb.llvm.org/python_reference/index.html
>
> Looks like an interesting SoC project to write useful lldb/gdb scripts to
> print internal structures for ease of debugging :-)
This seems like a very complicated mechanism of substituting for a
very simple patch. Your LLDB script is about the same number of lines
as Ashutosh's patch and only works for people who use LLDB. Plus,
once you write it, you've got to enter the Python interpreter to use
it and then run three more lines of code that aren't easy to remember.
In contrast, with Ashutosh's proposed patch, you just write:
p bms_to_char(bms_object)
...and you're done. Now, I grant that his approach bloats the binary
and yours does not, but nobody complains about pprint() bloating the
binary. If that's actually an issue people are really concerned about
then let's just reject this and Ashutosh can patch his local copy. If
that's not a serious problem then let's take the patch. If anything,
I think this discussion shows that LLDB macros are probably too much
of a pain to be seriously considered for everyday use, unless perhaps
you're the sort of person who plans to spend the day inside the Python
shell anyway.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 8:45 PM, Alvaro Herrera wrote: > Tom Lane wrote: >> Ashutosh Bapat writes: >> > While working on partition-wise join, I had to examine Relids objects >> > many times. Printing the Bitmapset::words[] in binary format and then >> > inferring the relids takes time and is error prone. >> >> FWIW, I generally rely on pprint() to look at planner data structures >> from the debugger. > > Also: > http://blog.pgaddict.com/posts/making-debugging-with-gdb-a-bit-easier > This may not address the issue directly, but it's probably very helpful. Thanks for the reference. I think this is something similar to what Pavan suggested in the mail thread. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
>> Alvaro Herrera writes: >>> I don't understand. Why don't you just use "call pprint(the bitmapset)" >>> in the debugger? >> >> Bitmapsets aren't Nodes, so pprint doesn't work directly on them. >> I usually find that I can pprint some node containing the value(s) >> I'm interested in, but maybe that isn't working for Ashutosh's >> particular case. that's right. > > There are many loose (ie, not inside any Node) Relids variables within the > optimizer code. Perhaps Ashutosh ended up needing to look at those a lot. that's right too. In joinrels.c for example we are manipulating Relids so many times. [ashutosh@ubuntu pg_head]grep bms_ src/backend/optimizer/path/joinrels.c | wc -l 69 There are many other instances of this in other optimizer and planner files. There are other places where we manipulate Bitmapsets. And not every Relids object computed is contained in a Node. So, pprint() doesn't help much. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Thu, Sep 15, 2016 at 2:58 PM, Ashutosh Bapat wrote: > On Wed, Sep 14, 2016 at 5:31 PM, Pavan Deolasee > wrote: >> The complete API reference is available here >> http://lldb.llvm.org/python_reference/index.html >> >> Looks like an interesting SoC project to write useful lldb/gdb scripts to >> print internal structures for ease of debugging :-) >> > > +1, if we can include something like that in the repository so as to > avoid every developer maintaining a script of his/her own. +1. Even if one finishes by doing manual modifications of some of them, it is always good to have a central point of reference. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 5:31 PM, Pavan Deolasee
wrote:
>
> On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee
> wrote:
>>
>>
>>
>> lately I'm using LVM debugger (which probably does not have something
>> equivalent),
>
>
> And I was so clueless about lldb's powerful scripting interface. For
> example, you can write something like this in bms_utils.py:
>
> import lldb
>
> def print_bms_members (bms):
> words = bms.GetChildMemberWithName("words")
> nwords = int(bms.GetChildMemberWithName("nwords").GetValue())
>
> ret = 'nwords = {0} bitmap: '.format(nwords,)
> for i in range(0, nwords):
> ret += hex(int(words.GetChildAtIndex(0, lldb.eNoDynamicValues,
> True).GetValue()))
>
> return ret
>
Thanks a lot for digging into it.
> And then do this while attached to lldb debugger:
>
> Process 99659 stopped
> * thread #1: tid = 0x59ba69, 0x0001090b012f
> postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673,
> queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
> frame #0: 0x0001090b012f
> postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673
>670 int wordnum,
>671 bitnum;
>672
> -> 673 if (x < 0)
>674 elog(ERROR, "negative bitmapset member not allowed");
>675 if (a == NULL)
>676 return bms_make_singleton(x);
> (lldb) script
> Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
from bms_utils import *
bms = lldb.frame.FindVariable ("a")
print print_bms_members(bms)
> nwords = 1 bitmap: 0x200
>
I can get that kind of output by command
p *bms
p/x (or p/b) *bms->words@(bms->nwords) in gdb.
But I can certainly extend the script you wrote above to print more
meaningful output similar to outBitmapset(). But then this would be
specific to LLVM. GDB too seems to have a similar interface
https://sourceware.org/gdb/wiki/PythonGdbTutorial, so I can probably
use above script with some modifications with GDB as well. Python
script will be easier to maintain as compared to maintaining a patch
that needs to be applied and compiled.
Said that, I am not sure if every debugger supported on every platform
we support has these features. Or may be developers work on only those
platforms which have such powerful debuggers, so it's ok.
Every debugger usually has much easier way to call a function and
print its output, so having a function like the one I have in the
patch makes things easy for all the debuggers and may be developers
not familiar with python.
>
> The complete API reference is available here
> http://lldb.llvm.org/python_reference/index.html
>
> Looks like an interesting SoC project to write useful lldb/gdb scripts to
> print internal structures for ease of debugging :-)
>
+1, if we can include something like that in the repository so as to
avoid every developer maintaining a script of his/her own.
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On 2016/09/15 0:04, Tom Lane wrote: > Alvaro Herrera writes: >> I don't understand. Why don't you just use "call pprint(the bitmapset)" >> in the debugger? > > Bitmapsets aren't Nodes, so pprint doesn't work directly on them. > I usually find that I can pprint some node containing the value(s) > I'm interested in, but maybe that isn't working for Ashutosh's > particular case. There are many loose (ie, not inside any Node) Relids variables within the optimizer code. Perhaps Ashutosh ended up needing to look at those a lot. Thanks, Amit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Tom Lane wrote: > Ashutosh Bapat writes: > > While working on partition-wise join, I had to examine Relids objects > > many times. Printing the Bitmapset::words[] in binary format and then > > inferring the relids takes time and is error prone. > > FWIW, I generally rely on pprint() to look at planner data structures > from the debugger. Also: http://blog.pgaddict.com/posts/making-debugging-with-gdb-a-bit-easier This may not address the issue directly, but it's probably very helpful. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Ashutosh Bapat writes: > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. FWIW, I generally rely on pprint() to look at planner data structures from the debugger. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Alvaro Herrera writes: > I don't understand. Why don't you just use "call pprint(the bitmapset)" > in the debugger? Bitmapsets aren't Nodes, so pprint doesn't work directly on them. I usually find that I can pprint some node containing the value(s) I'm interested in, but maybe that isn't working for Ashutosh's particular case. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
Ashutosh Bapat wrote: > Hi All, > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. Instead I wrote a > function bms_to_char() which allocates a StringInfo, calls > outBitmapset() to decode Bitmapset as a set of integers and returns > the string. In order to examine a Relids object all one has to do is > execute 'p bms_to_char(bms_object) under gdb. I don't understand. Why don't you just use "call pprint(the bitmapset)" in the debugger? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 3:46 PM, Pavan Deolasee
wrote:
>
>
> lately I'm using LVM debugger (which probably does not have something
> equivalent),
>
And I was so clueless about lldb's powerful scripting interface. For
example, you can write something like this in bms_utils.py:
import lldb
def print_bms_members (bms):
words = bms.GetChildMemberWithName("words")
nwords = int(bms.GetChildMemberWithName("nwords").GetValue())
ret = 'nwords = {0} bitmap: '.format(nwords,)
for i in range(0, nwords):
ret += hex(int(words.GetChildAtIndex(0, lldb.eNoDynamicValues,
True).GetValue()))
return ret
And then do this while attached to lldb debugger:
Process 99659 stopped
* thread #1: tid = 0x59ba69, 0x0001090b012f
postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at
bitmapset.c:673, queue = 'com.apple.main-thread', stop reason = breakpoint
1.1
frame #0: 0x0001090b012f
postgres`bms_add_member(a=0x7fe60a0351f8, x=10) + 15 at bitmapset.c:673
670 int wordnum,
671 bitnum;
672
-> 673 if (x < 0)
674 elog(ERROR, "negative bitmapset member not allowed");
675 if (a == NULL)
676 return bms_make_singleton(x);
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> from bms_utils import *
>>> bms = lldb.frame.FindVariable ("a")
>>> print print_bms_members(bms)
nwords = 1 bitmap: 0x200
The complete API reference is available here
http://lldb.llvm.org/python_reference/index.html
Looks like an interesting SoC project to write useful lldb/gdb scripts to
print internal structures for ease of debugging :-)
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] Printing bitmap objects in the debugger
On Wed, Sep 14, 2016 at 3:43 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > Hi All, > While working on partition-wise join, I had to examine Relids objects > many times. Printing the Bitmapset::words[] in binary format and then > inferring the relids takes time and is error prone. Instead I wrote a > function bms_to_char() which allocates a StringInfo, calls > outBitmapset() to decode Bitmapset as a set of integers and returns > the string. In order to examine a Relids object all one has to do is > execute 'p bms_to_char(bms_object) under gdb. > > Can we not do this as gdb macros? My knowledge is rusty in this area and lately I'm using LVM debugger (which probably does not have something equivalent), but I believe gdb allows you to write useful macros. As a bonus, you can then use them even for inspecting core files. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
