Re: [HACKERS] Gsoc2012 Idea --- Social Network database schema
The implementation seems to be done quite fully. There is even a patch file. Why is the implementation not added into the release of Postgres? As so much has already being done, what could I do in this case for the Gsoc? On Sun, Mar 18, 2012 at 6:38 PM, Daniel Farina wrote: > On Sat, Mar 17, 2012 at 8:48 PM, HuangQi wrote: > > About the second topic, so currently TABLESAMPLE is not implemented > > inside Postgres? I didn't see this query before, but I googled it just > now > > and the query seems very weird and > > interesting. > http://www.fotia.co.uk/fotia/DY.18.TheTableSampleClause.aspx > > Still, do you have any mail thread talking about this? > > I think there may be a few, but there's a nice implementation plan > discussed by Neil Conway and written into slides from a few years ago: > > http://www.pgcon.org/2007/schedule/attachments/9-Introduction_to_Hacking_PostgreSQL_Neil_Conway.pdf > > He also had his implementation, although at this point some of the > bitrot will be intense: > > http://www.neilconway.org/talks/hacking/ > > I also seem to remember writing this (to some degree) as a student as > part of a class project, so a full-blown production implementation in > a summer sounds reasonable, unless someone has thought more about this > and ran into some icebergs. I'm not sure exactly what the blockers > were to this being committed back in 2007 (not to suggest there > weren't any). > > I haven't thought enough about skipscan, but there a number more > unknowns there to me... > > -- > fdr > -- Best Regards Huang Qi Victor
Re: [HACKERS] Gsoc2012 Idea --- Social Network database schema
(Sorry, Daniel. Forgot to cc pgsql-hackers.) Hi, Daniel Thanks a lot for your response. As I can see for now, in my FYP, as the acyclic schema has the property that it has a join tree. I will check how many join trees it has and investigate any best option for the RSN schema. If it does have, I will modify the planner to just use this join tree if it detects a RSN database, then no need to use System R to search through all possible join trees. The implementation is narrow to only RSN schema. It might be too constraint for Postgres as a generic database system. For 'loose index join', I googled it and find the two sites useful to understand. http://dev.mysql.com/doc/refman/5.5/en/loose-index-scan.html http://wiki.postgresql.org/wiki/Loose_indexscan This is implemented in MySQL already, while Postgres only emulates the access method. Do you have any mail thread talking about the current design and progress? About the second topic, so currently TABLESAMPLE is not implemented inside Postgres? I didn't see this query before, but I googled it just now and the query seems very weird and interesting. http://www.fotia.co.uk/fotia/DY.18.TheTableSampleClause.aspx Still, do you have any mail thread talking about this? Thanks. -- Best Regards Huang Qi Victor
[HACKERS] Gsoc2012 Idea --- Social Network database schema
Hi, all I am a student of Computer Science and Applied Math in a university in Singapore. I'm planning to join Google Summer Code 2012 on PostgreSQL. It's quite an honor to join the postgresql hacker community. I have some postgresql developing experience while doing my school project. I'm doing Final Year Project(FYP) in this year, which is focusing on database and will be implemented in postgresql. My prof worked with some PhD students on social network database. They designed a database schema, called RSN (relationship on social network). My FYP is about studying the properties of RSN and try to implement more efficient optimisation algorithms for database on RSN. Currently I'm having some ideas on RSN. This schema reveals a property called acyclic property, and it preserves some special join property. The acyclic database schema was well studied, and there are many papers on it. I expect full working on this topic in three months would be enough to get idea as well as implementation. Please help me check whether this could become a topic for Postgres community. It focuses on a particular schema of social network, and this schema, to tell the truth, is still not quite mature yet. If this topic is not possible, I would like to check some other topics on query processing and optimisation, and doing the project together with my FYP in this summer. If you have some good idea on the proper topic, please recommand to me. I'm quite glad if you could offer me some advices. Thanks a lot for your help! -- Best Regards Huang Qi Victor
Re: [HACKERS] [Hackers]Backend quey plan process
On 28 June 2011 22:36, Alvaro Herrera wrote: > Excerpts from HuangQi's message of lun jun 27 23:56:22 -0400 2011: > > > BTW, which email system are you using to send to postgres mailing > list? > > As you can keep the top-answering and maintain the title of your email > with > > [hackers] in front, my gmail can not help on that. For this email, I just > > add by hand. > > You shouldn't add the [Hackers] prefix yourself. The mail list server > will add it automatically for you. Gmail deals with this perfectly > fine. > > -- > Álvaro Herrera > The PostgreSQL Company - Command Prompt, Inc. > PostgreSQL Replication, Consulting, Custom Development, 24x7 support > I see. Thanks. -- Best Regards Huang Qi Victor
[HACKERS] [Hackers]Backend quey plan process
Hi, I've been tracing the data structure in the query plan process for a while. But then I found the data structure manipulation is really so confusing. Could some guy tell me where could I find any guide on how to figure out the process and data structure usage? Is there any good resource helping us read the code? BTW, which email system are you using to send to postgres mailing list? As you can keep the top-answering and maintain the title of your email with [hackers] in front, my gmail can not help on that. For this email, I just add by hand. Thank you a lot for your help. -- Best Regards Huang Qi Victor
Re: [HACKERS] debugging tools inside postgres
On 24 June 2011 23:21, Tom Lane wrote: > Shigeru Hanada writes: > > (2011/06/24 15:35), HuangQi wrote: > >> I'm trying to debug a modification for the query planner. But I found it > >> seems the data structure of my planned query is incorrect. I was trying > to > >> print out the data structure by use the "p" command in gdb which is > quite > >> inconvenient and takes time. May I know is there any embedded function > in > >> postgres to print out the node data structures, or any other plan > related > >> data structures? Thanks. > > > I think nodeToString() would help. > > For interactive use in gdb, I generally do > >call pprint(..node pointer..) > > which prints to the postmaster log. > >regards, tom lane > Thanks, Tom, this call pprint() works very nice. -- Best Regards Huang Qi Victor
Re: [HACKERS] debugging tools inside postgres
2011/6/24 Shigeru Hanada > (2011/06/24 15:35), HuangQi wrote: > > Hi, > > I'm trying to debug a modification for the query planner. But I found > it > > seems the data structure of my planned query is incorrect. I was trying > to > > print out the data structure by use the "p" command in gdb which is quite > > inconvenient and takes time. May I know is there any embedded function in > > postgres to print out the node data structures, or any other plan related > > data structures? Thanks. > > I think nodeToString() would help. This function converts node tree > recursively into a string, and it's applicable for any Node-derived > object, such as Expr, Var and Const. > > ex) >elog(DEBUG1, "%s", nodeToString(plan)); > > Regards, > -- > Shigeru Hanada > Hi, I don't know why but when I am debugging the query evaluation, the elog function can not output to shell. -- Best Regards Huang Qi Victor
[HACKERS] debugging tools inside postgres
Hi, I'm trying to debug a modification for the query planner. But I found it seems the data structure of my planned query is incorrect. I was trying to print out the data structure by use the "p" command in gdb which is quite inconvenient and takes time. May I know is there any embedded function in postgres to print out the node data structures, or any other plan related data structures? Thanks. -- Best Regards Huang Qi Victor
Re: [HACKERS] gdb with postgres
On 6 June 2011 21:57, Kevin Grittner wrote: > HuangQi wrote: > > >>> (gdb) b qp_add_paths_to_joinrel > >>> Breakpoint 1 at 0x1a6744: file joinpath.c, line 67. > >>> (gdb) attach 23903 > > > If I enter c, gdb will directly finish executing this process and > > current query will finish. > > Are you absolutely sure that running your query will result in a > call to this function? > > -Kevin > Thanks guys for your idea. I found the solution. Because after made some change in postgres and make them, I didn't stop the server first. Now I stop it and start after reinstall, then using gdb is just fine. Thanks for your ideas. -- Best Regards Huang Qi Victor
Re: [HACKERS] gdb with postgres
If I enter c, gdb will directly finish executing this process and current query will finish. Furthermore, if I enter next query, gdb will not debug it and stay in continue status. On 6 June 2011 21:44, Robert Haas wrote: > On Mon, Jun 6, 2011 at 3:43 AM, HuangQi wrote: > > Hi, > >I was using gdb to debug postgres. In order to debug the backend of > > running query, I start postgres first and use "select * from > > pg_backend_pid()" to ask for backend pid. Then I start gdb in another > bash > > window with "gdb postgres" and attach the pid obtained above and set the > > breakpoint. Then I run the query from the first window. However, the > > debugging precess which is shown below is not going to the breakpoint. I > > tried many different breakpoints, but it always start from the 305 > > client_read_ended(). > > > > GNU gdb 6.6 > > Copyright (C) 2006 Free Software Foundation, Inc. > > GDB is free software, covered by the GNU General Public License, and you > are > > welcome to change it and/or distribute copies of it under certain > > conditions. > > Type "show copying" to see the conditions. > > There is absolutely no warranty for GDB. Type "show warranty" for > details. > > This GDB was configured as "sparc-sun-solaris2.10"... > > (gdb) b qp_add_paths_to_joinrel > > Breakpoint 1 at 0x1a6744: file joinpath.c, line 67. > > (gdb) attach 23903 > > Attaching to program `/usrlocal/pgsql/bin/postgres', process 23903 > > Retry #1: > > Retry #2: > > Retry #3: > > Retry #4: > > [New LWP 1] > > 0xff0cbaa4 in _rt_boot () from /usr/lib/ld.so.1 > > (gdb) n > > Single stepping until exit from function _rt_boot, > > which has no line number information. > > secure_read (port=0x4a7760, ptr=0x455948, len=8192) at be-secure.c:305 > > 305 client_read_ended(); > > (gdb) n > > pq_recvbuf () at pqcomm.c:767 > > 767 if (r < 0) > > (gdb) n > > 769 if (errno == EINTR) > > (gdb) n > > 782 if (r == 0) > > (gdb) n > > 788 return EOF; > > (gdb) n > > 791 PqRecvLength += r; > > (gdb) > > Any one know what is going wrong? BTW, as you can see, the system is > > solaris. > > Perhaps you want "c" for "continue" rather than "n" for "next". > > -- > Robert Haas > EnterpriseDB: http://www.enterprisedb.com > The Enterprise PostgreSQL Company > -- Best Regards Huang Qi Victor
Re: [HACKERS] gdb with postgres
Sorry, but recompile with this flag still doesn't work. On 6 June 2011 15:46, Pavan Deolasee wrote: > On Mon, Jun 6, 2011 at 1:13 PM, HuangQi wrote: > > Hi, > >I was using gdb to debug postgres. In order to debug the backend of > > running query, I start postgres first and use "select * from > > pg_backend_pid()" to ask for backend pid. Then I start gdb in another > bash > > window with "gdb postgres" and attach the pid obtained above and set the > > breakpoint. Then I run the query from the first window. However, the > > debugging precess which is shown below is not going to the breakpoint. I > > tried many different breakpoints, but it always start from the 305 > > client_read_ended(). > > > > Please compile with -O0 -g flags to see all the debug symbols. > > CFLAGS="-O0 -g" ./configure --enable-debug > > Thanks, > Pavan > > -- > Pavan Deolasee > EnterpriseDB http://www.enterprisedb.com > -- Best Regards Huang Qi Victor
[HACKERS] gdb with postgres
Hi, I was using gdb to debug postgres. In order to debug the backend of running query, I start postgres first and use "select * from pg_backend_pid()" to ask for backend pid. Then I start gdb in another bash window with "gdb postgres" and attach the pid obtained above and set the breakpoint. Then I run the query from the first window. However, the debugging precess which is shown below is not going to the breakpoint. I tried many different breakpoints, but it always start from the 305 client_read_ended(). GNU gdb 6.6 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "sparc-sun-solaris2.10"... (gdb) b qp_add_paths_to_joinrel Breakpoint 1 at 0x1a6744: file joinpath.c, line 67. (gdb) attach 23903 Attaching to program `/usrlocal/pgsql/bin/postgres', process 23903 Retry #1: Retry #2: Retry #3: Retry #4: [New LWP 1] 0xff0cbaa4 in _rt_boot () from /usr/lib/ld.so.1 (gdb) n Single stepping until exit from function _rt_boot, which has no line number information. secure_read (port=0x4a7760, ptr=0x455948, len=8192) at be-secure.c:305 305 client_read_ended(); (gdb) n pq_recvbuf () at pqcomm.c:767 767 if (r < 0) (gdb) n 769 if (errno == EINTR) (gdb) n 782 if (r == 0) (gdb) n 788 return EOF; (gdb) n 791 PqRecvLength += r; (gdb) Any one know what is going wrong? BTW, as you can see, the system is solaris. -- Best Regards Huang Qi Victor
Re: [HACKERS] Hacking gram.y Error syntax error at or near "MERGEJOIN"
Thanks all for your ideas. Though wired, I reinstalled the Postgres and tried again. This error message disappears. The parser now works good. On 3 June 2011 01:13, Alvaro Herrera wrote: > Excerpts from HuangQi's message of jue jun 02 11:17:21 -0400 2011: > > Hi, thanks a lot for your ideas. But I've done all these things. I've > > checked the gram.y and kwlist.h files many times but can not find what's > > wrong. So is there any possibility that the problem comes from something > > after parser, though it seems it should comes from parser? > > If you want more input, post the patch. > > -- > Álvaro Herrera > The PostgreSQL Company - Command Prompt, Inc. > PostgreSQL Replication, Consulting, Custom Development, 24x7 support > -- Best Regards Huang Qi Victor
Re: [HACKERS] Hacking gram.y Error syntax error at or near "MERGEJOIN"
Hi, thanks a lot for your ideas. But I've done all these things. I've checked the gram.y and kwlist.h files many times but can not find what's wrong. So is there any possibility that the problem comes from something after parser, though it seems it should comes from parser? On 2 June 2011 21:14, Heikki Linnakangas < heikki.linnakan...@enterprisedb.com> wrote: > On 02.06.2011 15:16, Andrew Dunstan wrote: > >> On 06/02/2011 03:28 AM, HuangQi wrote: >> >>> Hi, >>> I'm doing a hacking project which hacks the gram.y file and some other >>> so that postgres can execute some query plan language to run a >>> particular plan. I did some modifications and try run it. It compiled >>> and installed successfully. But when I was trying to run a MergeJoin >>> operation with keyword MERGEJOIN, it says "ERROR: syntax error at or >>> near 'MERGEJOIN'". It seems postgres didn't recognize this keyword. Is >>> it caused by any error inside gram.y? Or how can I trace this error? >>> >> >> Without seeing your grammar changes and the statement you're trying to >> execute it's pretty much impossible to tell. >> > > Yeah. One more guess is that you didn't include the keyword in the keyword > list at the end of gram.y. There's script to check various keyword-related > things at src/tools/check_keywords.pl > > -- > Heikki Linnakangas > EnterpriseDB http://www.enterprisedb.com > -- Best Regards Huang Qi Victor
Re: [HACKERS] Hacking gram.y Error syntax error at or near "MERGEJOIN"
Yes, I've also added mergejoin to kwlist.h as unreserved_keyword. In order to build AST, I also keep a new node struct in planoperators.h and a new node tag inside nodes.h. On 2 June 2011 15:49, Pavel Stehule wrote: > Hello > > you should to add new keywords to > postgresql/src/include/parser/kwlist.h (depends on pg version) > > attention - keywords must be in alphabet order > > Regards > > Pavel Stehule > > 2011/6/2 HuangQi : > > Hi, > > I'm doing a hacking project which hacks the gram.y file and some > other > > so that postgres can execute some query plan language to run a particular > > plan. I did some modifications and try run it. It compiled and installed > > successfully. But when I was trying to run a MergeJoin operation with > > keyword MERGEJOIN, it says "ERROR: syntax error at or near 'MERGEJOIN'". > It > > seems postgres didn't recognize this keyword. Is it caused by any error > > inside gram.y? Or how can I trace this error? > > > > -- > > Best Regards > > Huang Qi Victor > > > -- Best Regards Huang Qi Victor
[HACKERS] Hacking gram.y Error syntax error at or near "MERGEJOIN"
Hi, I'm doing a hacking project which hacks the gram.y file and some other so that postgres can execute some query plan language to run a particular plan. I did some modifications and try run it. It compiled and installed successfully. But when I was trying to run a MergeJoin operation with keyword MERGEJOIN, it says "ERROR: syntax error at or near 'MERGEJOIN'". It seems postgres didn't recognize this keyword. Is it caused by any error inside gram.y? Or how can I trace this error? -- Best Regards Huang Qi Victor
