Re: [HACKERS] IndexBuild Function call fcinfo cannot access memory
Dear Hackers, I have solved this problem. I just want to post my answer here for people who want to write their own backend index access method. When we write a backend index access method (Let's call this method ABC), we need to implement two interfaces: ABCbuild, ABCbuildcallback. ABCbuild is the main index initialization function and it will make some preparation and call (not directly call, ask Postgres to call) ABCbuildcallback function per each parent table tuple. For a given parent table tuple, ABCbuildcallback will generate the corresponding index entry and insert it into the index main body. When all parent table tuples have been processed by ABCbuildcallback, Postgres will return the control back to ABCbuild. As we can see, ABCbuild and ABCbuildcallback need to communicate with each other to transfer some information. This information is stored in a data structure called Struct ABCbuildstate. This ABCbuildstate is first initialized in ABCbuild and then passed to ABCbuildcallback. NOTE THAT: this Struct ABCbuildstate has to be initialized by palloc() in ABCbuild *because ABCbuild is not directly calling ABCbuildcallback*. My mistake is that I initialize/declare this ABCbuildstate struct by using regular C like this "ABCbuildstate my ABCbuildstate;" This will lead to segmentation fault sent by OS. *However, this wrong declare method still works when we add some unit SQL script in Postgres regression test (execute "make check"). We still are able to build ABC index "create index ABC_idx on ... using ABC" in Postgres regression test script. *But it will make us fail at normal Postgres server and general debug methods, such as set breakpoint or print debug information, cannot find this error because they even cannot reach the ABCbuild function. I believe there are some special settings in Postgres regression test program "src/test/regress/pg_regress.c" but I haven't found it. *These special settings will make Postgres regression test tolerates some memory errors. * Hope this will help you! Thanks, Jia Yu On Mon, Jan 23, 2017 at 1:40 PM, Jia Yu <jiayu198...@gmail.com> wrote: > Dear hackers, > > Currently, I am developing a backend index access method for my research > project. > > I built corresponding routines such as MyIndexbuild, MyIndexInsert, and so > on and put them in "src/backend/access/hippo" (hippo is my index's name). I > also added new entries in corresponding catalog files: > > src/include/catalog/pg_am.h > src/include/catalog/pg_amop.h > src/include/catalog/pg_amproc.h > src/include/catalog/pg_opclass.h > src/include/catalog/pg_opfamily.h > src/include/catalog/pg_proc.h > > Now I can successfully build/query/update my new index by executing PG > source code regression test (make check). Everything looks fine. > > > However, these methods don't work in the normal PG server. It gave me > "segmentation fault" > > I entered this SQL command in psql: > create index hippo_idx on hippo_tbl using hippo(id2); > > Then got: > Segmentation fault (core dumped) > > Here is my backtrace. It looks like I cannot access fcinfo. Can you help > me about this? Or just some hints? I have been struggling with this problem > for weeks. > > Thank you all in advance! > > > #0 0x004673d2 in hippobuild ( > fcinfo= 0x7fff3092c188>) at hippo.c:154 > #1 0x00956e64 in OidFunctionCall3Coll (functionId=5005, > collation=0, > arg1=140162511383536, arg2=140162511390256, arg3=19885792) at > fmgr.c:1649 > #2 0x00555809 in index_build (heapRelation=0x7f7a20b3abf0, > indexRelation=0x7f7a20b3c630, indexInfo=0x12f6ee0, isprimary=0 '\000', > isreindex=0 '\000') at index.c:2025 > #3 0x00554551 in index_create (heapRelation=0x7f7a20b3abf0, > indexRelationName=0x12f9378 "hippo_idx", indexRelationId=20668, > relFileNode=0, indexInfo=0x12f6ee0, indexColNames=0x12f7370, > accessMethodObjectId=5000, tableSpaceId=0, > collationObjectId=0x12f7990, > classObjectId=0x12f79b0, coloptions=0x12f79d0, reloptions=19895952, > isprimary=0 '\000', isconstraint=0 '\000', deferrable=0 '\000', > initdeferred=0 '\000', allow_system_table_mods=0 '\000', > skip_build=0 '\000', concurrent=0 '\000', is_internal=0 '\000', > if_not_exists=0 '\000') at index.c:1100 > #4 0x0062727b in DefineIndex (relationId=12476, stmt=0x12f6ff8, > indexRelationId=0, is_alter_table=0 '\000', check_rights=1 '\001', > skip_build=0 '\000', quiet=0 '\000') at indexcmds.c:606 > #5 0x008077a6 in ProcessUtilitySlow (parsetree=0x127c740, > queryString=0x127ba48 "create index hippo_idx on hippo_tbl using > hippo(id2) with (density=20);\n", context=PRO
Re: [HACKERS] IndexBuild Function call fcinfo cannot access memory
Dear Hackers,
Sorry for bothering you again!
I have migrated my index method from PG 9.5 to PG 9.6.1 as Tom suggested.
The same problem still exists. The index works fine if I just call it
through my new SQL script in regression test (call "make check").
I am doubting whether this is caused by some security issues.
I compiled the code on PG 9.6.1 with flag -O0. It gave me this same error.
The new backtraces are as this:
---
#0 0x0046bfbd in hippobuild (heap=, index=, indexInfo=) at hippo.c:204
#1 0x0055d1c9 in index_build (heapRelation=0x7f0f3bdfea88,
indexRelation=0x7f0f3be002b0, indexInfo=0x2e1d180, isprimary=0 '\000',
isreindex=0 '\000') at index.c:2021
#2 0x0055beda in index_create (heapRelation=0x7f0f3bdfea88,
indexRelationName=0x2ec2378 "hippo_idx", indexRelationId=20695,
relFileNode=0, indexInfo=0x2e1d180, indexColNames=0x2ec0370,
accessMethodObjectId=9000, tableSpaceId=0, collationObjectId=0x2ec0938,
classObjectId=0x2ec0958,
coloptions=0x2ec0978, reloptions=49030800, isprimary=0 '\000',
isconstraint=0 '\000', deferrable=0 '\000', initdeferred=0 '\000',
allow_system_table_mods=0 '\000', skip_build=0 '\000', concurrent=0 '\000',
is_internal=0 '\000', if_not_exists=0 '\000') at index.c:1099
#3 0x00632d7a in DefineIndex (relationId=12503, stmt=0x2e1d298,
indexRelationId=0, is_alter_table=0 '\000', check_rights=1 '\001',
skip_build=0 '\000', quiet=0 '\000') at indexcmds.c:651
#4 0x0083660e in ProcessUtilitySlow (parsetree=0x2e7ba00,
queryString=0x2e7acc8 "create index hippo_idx on hippo_tbl using hippo(id2)
with (density=20);\n", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
dest=0xe68c60 , completionTag=0x7ffc613bf280 "") at
utility.c:1274
#5 0x00835a98 in standard_ProcessUtility (parsetree=0x2e7ba00,
queryString=0x2e7acc8 "create index hippo_idx on hippo_tbl using hippo(id2)
with (density=20);\n", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
dest=0xe68c60 , completionTag=0x7ffc613bf280 "") at
utility.c:907
#6 0x00834b71 in ProcessUtility (parsetree=0x2e7ba00,
queryString=0x2e7acc8 "create index hippo_idx on hippo_tbl using hippo(id2)
with (density=20);\n", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
dest=0xe68c60 , completionTag=0x7ffc613bf280 "") at
utility.c:336
#7 0x00833c1b in PortalRunUtility (portal=0x2eba198,
utilityStmt=0x2e7ba00, isTopLevel=1 '\001', setHoldSnapshot=0 '\000',
dest=0xe68c60 , completionTag=0x7ffc613bf280 "") at
pquery.c:1193
#8 0x00833e2e in PortalRunMulti (portal=0x2eba198, isTopLevel=1
'\001', setHoldSnapshot=0 '\000', dest=0xe68c60 ,
altdest=0xe68c60 , completionTag=0x7ffc613bf280 "") at
pquery.c:1342
#9 0x0083332f in PortalRun (portal=0x2eba198,
count=9223372036854775807, isTopLevel=1 '\001', dest=0xe68c60 ,
altdest=0xe68c60 , completionTag=0x7ffc613bf280 "") at
pquery.c:815
#10 0x0082d05b in exec_simple_query (query_string=0x2e7acc8 "create
index hippo_idx on hippo_tbl using hippo(id2) with (density=20);\n") at
postgres.c:1094
#11 0x00831479 in PostgresMain (argc=6, argv=0x2df5a70,
dbname=0x2dfd3c0 "postgres", username=0x2dfd3c0 "postgres") at
postgres.c:4070
#12 0x006e7832 in main (argc=6, argv=0x2df5a70) at main.c:224
---
The problematic line is at Hippo.c Line 204. Here is the code of that line.
It is the regular routine required by designing an index access method. It
will build the index.
--
/*
*This function initializes the entire Hippo. It will call buildcallback
many times.
*/
IndexBuildResult *
hippobuild(Relation heap, Relation index, IndexInfo *indexInfo)
*Line 204:*{
IndexBuildResult *result;
double reltuples;
HippoBuildState buildstate;
Buffer buffer;
Datum *histogramBounds;
int histogramBoundsNum,i;
HeapTuple heapTuple;
Page page;/* Initial page */
Size pageSize;
AttrNumber attrNum;
--
This is my SQL script which works fine when running in PG regression test
parallel_schedule. But it fails at normal PG server.
--
create table hippo_tbl(id int8, id2 int8, payload text);
insert into hippo_tbl(id, id2, payload) select i, random()*100,
repeat('a', 100) from generate_series (1,100) i;
Alter table hippo_tbl alter column id2 set statistics 400;
Analyze hippo_tbl;
create index hippo_idx on hippo_tbl using hippo(id2) with (density=20);
select count(*) from hippo_tbl where id2>10 and id2 <101000;
insert into hippo_tbl(id, id2, payload) select i, 19, repeat('a', 100)
from generate_series (1, 1000) i;
select count(*) from hippo_tbl where id2>10 and id2 <101000;
delete from hippo_tbl where id2>10 and id2 <101000;
VACUUM;
select count(*) from hippo_tbl where id2>10 and id2 <101000;
drop inde
Re: [HACKERS] IndexBuild Function call fcinfo cannot access memory
Dear Tom,
BTW, my current implementation is on PG 9.5...
Thanks,
Jia Yu
Jia Yu, Ph.D. Student
Computer Science
Arizona State University <http://www.asu.edu/>
Reach me via Jia Yu's Homepage <http://www.public.asu.edu/~jiayu2/> | GitHub
Repositories <https://github.com/jiayuasu>
On Mon, Jan 23, 2017 at 5:12 PM, Jia Yu <jiayu198...@gmail.com> wrote:
> Dear Tom,
>
> Thanks for your reply!
>
> Actually, I already compiled the source code with "-O0" and then got those
> backtraces.
>
> The hippobuild function just starts building an index access method and it
> will call hippobuildcallback to iterate each parent table tuple in its
> function body.
>
> What makes me fell very confused is that the same code works when I call
> "make check" to run regression test (testing my index) but fails at running
> normal PG server.
>
> Here are the lines before and after the problematic line hippo.c 154. It
> is the first line of this function and did nothing.
>
> On the other hand, I will try to adjust my code to fit in PG 9.6 as you
> suggested.
>
> --
> static void hippobuildcallback()
> {
> .
> .
> .
> *Line 148* }
> *Line 149* /*
> *Line 150* *This function initializes the entire Hippo. It will call
> buildcallback many times.
> *Line 151* */
> *Line 152* Datum
> *Line 153* hippobuild(PG_FUNCTION_ARGS)
> *Line 154* {
> *Line 155* /*
> * This is some routine declarations for variables
>*/
> Relation heap = (Relation) PG_GETARG_POINTER(0);
> Relation index = (Relation) PG_GETARG_POINTER(1);
> IndexInfo *indexInfo = (IndexInfo *) PG_GETARG_POINTER(2);
> IndexBuildResult *result;
> double reltuples;
> HippoBuildState buildstate;
> Buffer buffer;
> ..
> ----------
>
> Any hints for this will be greatly appreciated!
>
>
> Thank you very much!
>
> Jia Yu
>
> On Mon, Jan 23, 2017 at 2:00 PM, Tom Lane <t...@sss.pgh.pa.us> wrote:
>
>> Jia Yu <jiayu198...@gmail.com> writes:
>> > However, these methods don't work in the normal PG server. It gave me
>> > "segmentation fault"
>> > ...
>> > Here is my backtrace. It looks like I cannot access fcinfo. Can you
>> help me
>> > about this? Or just some hints? I have been struggling with this problem
>> > for weeks.
>>
>> What's the problematic line (hippo.c:154) doing? What's before
>> that in the hippobuild function?
>>
>> I wouldn't put much stock in the "Cannot access memory" message ... that
>> just means gdb is confused, which it frequently is. But possibly
>> building with -O0 would make it less confused.
>>
>> BTW, I gather from the reference to OidFunctionCall3Coll that you're
>> using PG 9.5 or older. It'd be a good idea to switch to 9.6, which has
>> noticeably cleaner indexam APIs that allow some compiler error checking
>> on those calls.
>>
>> regards, tom lane
>>
>
>
Re: [HACKERS] IndexBuild Function call fcinfo cannot access memory
Dear Tom,
Thanks for your reply!
Actually, I already compiled the source code with "-O0" and then got those
backtraces.
The hippobuild function just starts building an index access method and it
will call hippobuildcallback to iterate each parent table tuple in its
function body.
What makes me fell very confused is that the same code works when I call
"make check" to run regression test (testing my index) but fails at running
normal PG server.
Here are the lines before and after the problematic line hippo.c 154. It is
the first line of this function and did nothing.
On the other hand, I will try to adjust my code to fit in PG 9.6 as you
suggested.
--
static void hippobuildcallback()
{
.
.
.
*Line 148* }
*Line 149* /*
*Line 150* *This function initializes the entire Hippo. It will call
buildcallback many times.
*Line 151* */
*Line 152* Datum
*Line 153* hippobuild(PG_FUNCTION_ARGS)
*Line 154* {
*Line 155* /*
* This is some routine declarations for variables
*/
Relation heap = (Relation) PG_GETARG_POINTER(0);
Relation index = (Relation) PG_GETARG_POINTER(1);
IndexInfo *indexInfo = (IndexInfo *) PG_GETARG_POINTER(2);
IndexBuildResult *result;
double reltuples;
HippoBuildState buildstate;
Buffer buffer;
..
--
Any hints for this will be greatly appreciated!
Thank you very much!
Jia Yu
On Mon, Jan 23, 2017 at 2:00 PM, Tom Lane <t...@sss.pgh.pa.us> wrote:
> Jia Yu <jiayu198...@gmail.com> writes:
> > However, these methods don't work in the normal PG server. It gave me
> > "segmentation fault"
> > ...
> > Here is my backtrace. It looks like I cannot access fcinfo. Can you help
> me
> > about this? Or just some hints? I have been struggling with this problem
> > for weeks.
>
> What's the problematic line (hippo.c:154) doing? What's before
> that in the hippobuild function?
>
> I wouldn't put much stock in the "Cannot access memory" message ... that
> just means gdb is confused, which it frequently is. But possibly
> building with -O0 would make it less confused.
>
> BTW, I gather from the reference to OidFunctionCall3Coll that you're
> using PG 9.5 or older. It'd be a good idea to switch to 9.6, which has
> noticeably cleaner indexam APIs that allow some compiler error checking
> on those calls.
>
> regards, tom lane
>
[HACKERS] IndexBuild Function call fcinfo cannot access memory
Dear hackers, Currently, I am developing a backend index access method for my research project. I built corresponding routines such as MyIndexbuild, MyIndexInsert, and so on and put them in "src/backend/access/hippo" (hippo is my index's name). I also added new entries in corresponding catalog files: src/include/catalog/pg_am.h src/include/catalog/pg_amop.h src/include/catalog/pg_amproc.h src/include/catalog/pg_opclass.h src/include/catalog/pg_opfamily.h src/include/catalog/pg_proc.h Now I can successfully build/query/update my new index by executing PG source code regression test (make check). Everything looks fine. However, these methods don't work in the normal PG server. It gave me "segmentation fault" I entered this SQL command in psql: create index hippo_idx on hippo_tbl using hippo(id2); Then got: Segmentation fault (core dumped) Here is my backtrace. It looks like I cannot access fcinfo. Can you help me about this? Or just some hints? I have been struggling with this problem for weeks. Thank you all in advance! #0 0x004673d2 in hippobuild ( fcinfo=) at hippo.c:154 #1 0x00956e64 in OidFunctionCall3Coll (functionId=5005, collation=0, arg1=140162511383536, arg2=140162511390256, arg3=19885792) at fmgr.c:1649 #2 0x00555809 in index_build (heapRelation=0x7f7a20b3abf0, indexRelation=0x7f7a20b3c630, indexInfo=0x12f6ee0, isprimary=0 '\000', isreindex=0 '\000') at index.c:2025 #3 0x00554551 in index_create (heapRelation=0x7f7a20b3abf0, indexRelationName=0x12f9378 "hippo_idx", indexRelationId=20668, relFileNode=0, indexInfo=0x12f6ee0, indexColNames=0x12f7370, accessMethodObjectId=5000, tableSpaceId=0, collationObjectId=0x12f7990, classObjectId=0x12f79b0, coloptions=0x12f79d0, reloptions=19895952, isprimary=0 '\000', isconstraint=0 '\000', deferrable=0 '\000', initdeferred=0 '\000', allow_system_table_mods=0 '\000', skip_build=0 '\000', concurrent=0 '\000', is_internal=0 '\000', if_not_exists=0 '\000') at index.c:1100 #4 0x0062727b in DefineIndex (relationId=12476, stmt=0x12f6ff8, indexRelationId=0, is_alter_table=0 '\000', check_rights=1 '\001', skip_build=0 '\000', quiet=0 '\000') at indexcmds.c:606 #5 0x008077a6 in ProcessUtilitySlow (parsetree=0x127c740, queryString=0x127ba48 "create index hippo_idx on hippo_tbl using hippo(id2) with (density=20);\n", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0xe19a40 , completionTag=0x7fff7819aae0 "") at utility.c:1260 #6 0x00806c30 in standard_ProcessUtility (parsetree=0x127c740, queryString=0x127ba48 "create index hippo_idx on hippo_tbl using hippo(id2) with (density=20);\n", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0xe19a40 , completionTag=0x7fff7819aae0 "") at utility.c:893 #7 0x00805d81 in ProcessUtility (parsetree=0x127c740, queryString=0x127ba48 "create index hippo_idx on hippo_tbl using hippo(id2) with (density=20);\n", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0xe19a40 , completionTag=0x7fff7819aae0 "") at utility.c:335 #8 0x00804dd3 in PortalRunUtility (portal=0x12f0d88, utilityStmt=0x127c740, isTopLevel=1 '\001', dest=0xe19a40 , completionTag=0x7fff7819aae0 "") at pquery.c:1187 #9 0x00804fa1 in PortalRunMulti (portal=0x12f0d88, isTopLevel=1 '\001', dest=0xe19a40 , altdest=0xe19a40 , completionTag=0x7fff7819aae0 "") at pquery.c:1318 #10 0x008044e5 in PortalRun (portal=0x12f0d88, count=9223372036854775807, isTopLevel=1 '\001', dest=0xe19a40 , altdest=0xe19a40 , completionTag=0x7fff7819aae0 "") at pquery.c:816 #11 0x007fe2af in exec_simple_query ( query_string=0x127ba48 "create index hippo_idx on hippo_tbl using hippo(id2) with (density=20);\n") at postgres.c:1104 #12 0x008025af in PostgresMain (argc=6, argv=0x1244a40, dbname=0x124c3a0 "postgres", username=0x124c3a0 "postgres") at postgres.c:4025 #13 0x006d3513 in main (argc=6, argv=0x1244a40) at main.c:219
