Re: [HACKERS] using arrays within structure in ECPG
Hi Michael, The problem of offsets seems to be universal. If there is a structure within structure. The offset to the members of inner structure should be the size of the outer structure and not size of inner structure. Applying this rule recursively, offset to the member of any nested structure, at whatever level of nesting it is, should be same as the size of the outermost structure. But the code as of now, is using the size of the immediate parent. None of these problems are caught in the regression because, whatever tests I have seen are not fetching more than one tuple into such complex structure. On Tue, Apr 1, 2014 at 4:34 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > Hi MIchael, > I tried to fix the offset problem. PFA the patch. It does solve the > problem of setting wrong offset in ECPGdo() call. > > But then there is problem of interpreting the result from server as an > array within array of structure. The problem is there is in > ecpg_get_data(). This function can not understand that the "field" is an > array of integers (or for that matter array of anything) and store all the > values in contiguous memory at the given address. > > > > On Thu, Mar 27, 2014 at 11:05 PM, Michael Meskes wrote: > >> On Mon, Mar 24, 2014 at 11:52:30AM +0530, Ashutosh Bapat wrote: >> > For all the members of struct employee, except arr_col, the size of >> array >> > is set to 14 and next member offset is set of sizeof (struct employee). >> But >> > for arr_col they are set to 3 and sizeof(int) resp. So, for the next row >> > onwards, the calculated offset of arr_col member would not coincide with >> > the real arr_col member's address. >> > >> > Am I missing something here? >> >> No, this looks like a bug to me. I haven't had time to look into the >> source codebut the offset definitely is off. >> >> Michael >> -- >> Michael Meskes >> Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) >> Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org >> Jabber: michael.meskes at gmail dot com >> VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL >> > > > > -- > Best Wishes, > Ashutosh Bapat > EnterpriseDB Corporation > The Postgres Database Company > -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
Re: [HACKERS] using arrays within structure in ECPG
On Wed, Apr 02, 2014 at 05:49:03PM +0530, Ashutosh Bapat wrote: > I have one more doubt, regarding offsets. > ... This is actually a very good question. Parts of this code are older than my work on ecpg, meaning they were already in version 0.1. It could very well be that with some changes over the years this test isn't needed anymore. The regression suite works without just nicely. To be honest, the whole type.c code needs a real rewrite to be better legible. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at gmail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] using arrays within structure in ECPG
I have one more doubt, regarding offsets. In ECPGdump_a_simple() we have code if (siz == NULL || strlen(siz) == 0 || strcmp(arrsize, "0") == 0 || strcmp(arrsize, "1") == 0) fprintf(o, "\n\t%s,%s,(long)%s,(long)%s,%s, ", get_type(type), variable, varcharsize, arrsize, offset); else fprintf(o, "\n\t%s,%s,(long)%s,(long)%s,%s, ", get_type(type), variable, varcharsize, arrsize, siz); If the caller has passed siz, it means that this variable is part of the a structure. Remember in dump_variables(), this function is called with struct_sizeof = NULL. So, once we know that siz != NULL and strlen(siz) != 0, it's evident that the simple variable we are dumping is part of a structure and hence we should be using "siz" instead of "offset". Why then we still check arrsize? In a case, where we are dumping a pointer to a structure, this code dumps each member with offset = size of that member, thus again corrupting memory, if there are more than one rows being saved through pointer. On Wed, Apr 2, 2014 at 3:10 PM, Michael Meskes wrote: > On Wed, Apr 02, 2014 at 09:33:15AM +0530, Ashutosh Bapat wrote: > > So, you are saying that we should try to catch such errors and report > > during pre-compile time. That's better than silently corrupting the data. > > Well, I think this goes without saying. > > Michael > -- > Michael Meskes > Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) > Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org > Jabber: michael.meskes at gmail dot com > VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL > -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
Re: [HACKERS] using arrays within structure in ECPG
On Wed, Apr 02, 2014 at 09:33:15AM +0530, Ashutosh Bapat wrote: > So, you are saying that we should try to catch such errors and report > during pre-compile time. That's better than silently corrupting the data. Well, I think this goes without saying. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at gmail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] using arrays within structure in ECPG
So, you are saying that we should try to catch such errors and report during pre-compile time. That's better than silently corrupting the data. On Tue, Apr 1, 2014 at 10:20 PM, Michael Meskes wrote: > Hi Ashutosh, > > > I tried to fix the offset problem. PFA the patch. It does solve the > > problem of setting wrong offset in ECPGdo() call. > > Thanks, looks correct to me. > > > But then there is problem of interpreting the result from server as an > > array within array of structure. The problem is there is in > > ecpg_get_data(). This function can not understand that the "field" is an > > array of integers (or for that matter array of anything) and store all > > the values in contiguous memory at the given address. > > I guess I know where that comes from, without actually looking at the > code, though. Nested arrays are not supported by ecpg and the > precompiler spits out an error message, just check preproc/type.c. > However, in your example you have the struct essantially sandwiched > between the arrays and the (too) simple check in that file doesn't > notice, but because the implementation is nevertheless lacking. > > I'm sorry, but this sounds like a missing feature bug. > > Michael > -- > Michael Meskes > Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) > Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org > Jabber: michael.meskes at gmail dot com > VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL > -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
Re: [HACKERS] using arrays within structure in ECPG
On Tue, Apr 1, 2014 at 11:50 AM, Michael Meskes wrote: > Hi Ashutosh, > >> I tried to fix the offset problem. PFA the patch. It does solve the >> problem of setting wrong offset in ECPGdo() call. > > Thanks, looks correct to me. > >> But then there is problem of interpreting the result from server as an >> array within array of structure. The problem is there is in >> ecpg_get_data(). This function can not understand that the "field" is an >> array of integers (or for that matter array of anything) and store all >> the values in contiguous memory at the given address. > > I guess I know where that comes from, without actually looking at the > code, though. Nested arrays are not supported by ecpg and the > precompiler spits out an error message, just check preproc/type.c. > However, in your example you have the struct essantially sandwiched > between the arrays and the (too) simple check in that file doesn't > notice, but because the implementation is nevertheless lacking. > > I'm sorry, but this sounds like a missing feature bug. Small aside: I've often wondered if the right long term approach is to abstract backend type code into a shared library that both the server and (optionally) the client would link with. That would make extending support to exotic types in ecpg much easier. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] using arrays within structure in ECPG
Hi Ashutosh, > I tried to fix the offset problem. PFA the patch. It does solve the > problem of setting wrong offset in ECPGdo() call. Thanks, looks correct to me. > But then there is problem of interpreting the result from server as an > array within array of structure. The problem is there is in > ecpg_get_data(). This function can not understand that the "field" is an > array of integers (or for that matter array of anything) and store all > the values in contiguous memory at the given address. I guess I know where that comes from, without actually looking at the code, though. Nested arrays are not supported by ecpg and the precompiler spits out an error message, just check preproc/type.c. However, in your example you have the struct essantially sandwiched between the arrays and the (too) simple check in that file doesn't notice, but because the implementation is nevertheless lacking. I'm sorry, but this sounds like a missing feature bug. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at gmail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] using arrays within structure in ECPG
Hi MIchael,
I tried to fix the offset problem. PFA the patch. It does solve the problem
of setting wrong offset in ECPGdo() call.
But then there is problem of interpreting the result from server as an
array within array of structure. The problem is there is in
ecpg_get_data(). This function can not understand that the "field" is an
array of integers (or for that matter array of anything) and store all the
values in contiguous memory at the given address.
On Thu, Mar 27, 2014 at 11:05 PM, Michael Meskes wrote:
> On Mon, Mar 24, 2014 at 11:52:30AM +0530, Ashutosh Bapat wrote:
> > For all the members of struct employee, except arr_col, the size of array
> > is set to 14 and next member offset is set of sizeof (struct employee).
> But
> > for arr_col they are set to 3 and sizeof(int) resp. So, for the next row
> > onwards, the calculated offset of arr_col member would not coincide with
> > the real arr_col member's address.
> >
> > Am I missing something here?
>
> No, this looks like a bug to me. I haven't had time to look into the
> source codebut the offset definitely is off.
>
> Michael
> --
> Michael Meskes
> Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
> Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
> Jabber: michael.meskes at gmail dot com
> VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL
>
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
diff --git a/src/interfaces/ecpg/preproc/type.c b/src/interfaces/ecpg/preproc/type.c
index 2982cb6..53983fe 100644
--- a/src/interfaces/ecpg/preproc/type.c
+++ b/src/interfaces/ecpg/preproc/type.c
@@ -296,21 +296,22 @@ ECPGdump_a_type(FILE *o, const char *name, struct ECPGtype * type, const int bra
type->u.element,
(ind_type == NULL) ? NULL : ((ind_type->type == ECPGt_NO_INDICATOR) ? ind_type : ind_type->u.element),
prefix, ind_prefix);
break;
default:
if (!IS_SIMPLE_TYPE(type->u.element->type))
base_yyerror("internal error: unknown datatype, please report this to ");
ECPGdump_a_simple(o, name,
type->u.element->type,
- type->u.element->size, type->size, NULL, prefix, type->u.element->counter);
+ type->u.element->size, type->size, struct_sizeof ? struct_sizeof : NULL,
+ prefix, type->u.element->counter);
if (ind_type != NULL)
{
if (ind_type->type == ECPGt_NO_INDICATOR)
{
char *str_neg_one = mm_strdup("-1");
ECPGdump_a_simple(o, ind_name, ind_type->type, ind_type->size, str_neg_one, NULL, ind_prefix, 0);
free(str_neg_one);
}
else
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] using arrays within structure in ECPG
On Mon, Mar 24, 2014 at 11:52:30AM +0530, Ashutosh Bapat wrote: > For all the members of struct employee, except arr_col, the size of array > is set to 14 and next member offset is set of sizeof (struct employee). But > for arr_col they are set to 3 and sizeof(int) resp. So, for the next row > onwards, the calculated offset of arr_col member would not coincide with > the real arr_col member's address. > > Am I missing something here? No, this looks like a bug to me. I haven't had time to look into the source codebut the offset definitely is off. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at gmail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] using arrays within structure in ECPG
On Mon, Mar 24, 2014 at 3:40 PM, Boszormenyi Zoltan wrote:
> 2014-03-24 07:22 keltezéssel, Ashutosh Bapat írta:
>
> Hi,
> I tried using integer array within a structure array in ECPG code. But
> it resulted in some garbage values being printed from the table. Here are
> the details,
>
> The ECPG program is attached (array_test.pgc). It tries to read the
> contents of table emp, whose structure and contents are as follows
> postgres=# \d+ emp
>Table "public.emp"
> Column | Type| Modifiers | Storage | Stats target |
> Description
>
> -+---+---+--+--+-
> empno | numeric(4,0) | | main | |
> ename | character varying | | extended | |
> job | character varying | | extended | |
> arr_col | integer[] | | extended | |
> Has OIDs: no
>
> postgres=# select * from emp;
> empno | ename | job | arr_col
> ---++-+
> 7900 | JAMES | CLERK | {1,2,7900}
> 7902 | FORD | ANALYST | {1,2,7902}
> 7934 | MILLER | CLERK | {1,2,7934}
> (3 rows)
>
> You will notice that the last element of the arr_col array is same as
> the empno of that row.
>
> The ECPG program tries to read the rows using FETCH in a structure emp
> defined as
> 15 struct employee {
> 16 int empno;
> 17char ename[11];
> 18char job[15];
> 19int arr_col[3];
> 20 };
>
> and then print the read contents as
> 39 /* Print members of the structure. */
> 40 for ( i = 0 ;i < 3; i++){
> 41 printf("empno=%d, ename=%s, job=%s, arr_col[2]=%d\n",
> emp[i].empno, emp[i].ename, emp[i].job, emp[i].arr_col[2]);
> 42
> 43 }
>
> But garbage values are printed
> [ashutosh@ubuntu repro]./array_test
>
> +++
> empno=7900, ename=JAMES, job=CLERK, arr_col[2]=1
> empno=2, ename=� , job=ANALYST, arr_col[2]=32767
> empno=7934, ename=MILLER, job=CLERK, arr_col[2]=1719202336
>
> Here are steps I have used to compile the ECPG program
> [ashutosh@ubuntu repro]make array_test
> ecpg -c -I/work/pg_head/build/include array_test.pgc
> cc -I/work/pg_head/build/include -g -c -o array_test.o array_test.c
> cc -g array_test.o -L/work/pg_head/build/lib -lecpg -lpq -o array_test
> rm array_test.o array_test.c
>
> where /work/pg_head/build is the directory containing the postgresql
> build (essentially argument to the --prefix option to configure).
>
> The programs compiles and links fine.
>
> Without the arr_col member, the program works fine. So, it seems to be a
> problem with array within structure array.
>
> In array_test.c I see that the ECPGdo statement corresponding to the
> FETCH command is as follows
> 87 /* Fetch multiple columns into one structure. */
> 88 { ECPGdo(__LINE__, 0, 1, NULL, 0, ECPGst_normal, "fetch 3 from
> cur1", ECPGt_EOIT,
> 89 ECPGt_int,&(emp->empno),(long)1,(long)14,sizeof( struct employee ),
> 90 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
> 91 ECPGt_char,&(emp->ename),(long)11,(long)14,sizeof( struct employee
> ),
> 92 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
> 93 ECPGt_char,&(emp->job),(long)15,(long)14,sizeof( struct employee ),
> 94 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
> 95 ECPGt_int,(emp->arr_col),(long)1,(long)3,sizeof(int),
> 96 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EORT);
>
> For all the members of struct employee, except arr_col, the size of
> array is set to 14 and next member offset is set of sizeof (struct
> employee). But for arr_col they are set to 3 and sizeof(int) resp. So, for
> the next row onwards, the calculated offset of arr_col member would not
> coincide with the real arr_col member's address.
>
> Am I missing something here?
>
>
> ECPG (I think intentionally) doesn't interpret or parse array fields.
> You need to pass a character array and parse the contents by yourself.
>
>
That doesn't seem to be the case with bare arrays (not included in anything
else). I added following lines to the program attached in my first mail,
and they worked perfectly fine.
47 /* Test only arrays */
48 EXEC SQL DECLARE cur2 CURSOR FOR select arr_col from emp;
49
50 EXEC SQL OPEN cur2;
51
52 EXEC SQL FETCH 1 FROM cur2 into :emp[0].arr_col;
53
54 printf("\n+++\n");
55
56 /* Print members of the array fetched. */
57 for ( i = 0 ;i < 3; i++)
58 {
59 printf("arr_col[%d] = %d\n", i, emp[0].arr_col[i]);
60 }
61 EXEC SQL CLOSE cur2;
Anyway, if that's a restriction, shouldn't there be an error during
compilation?
> Best regards,
> Zoltán Böszörményi
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>
>
>
>
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporat
Re: [HACKERS] using arrays within structure in ECPG
2014-03-24 07:22 keltezéssel, Ashutosh Bapat írta:
Hi,
I tried using integer array within a structure array in ECPG code. But it resulted in
some garbage values being printed from the table. Here are the details,
The ECPG program is attached (array_test.pgc). It tries to read the contents of table
emp, whose structure and contents are as follows
postgres=# \d+ emp
Table "public.emp"
Column | Type| Modifiers | Storage | Stats target | Description
-+---+---+--+--+-
empno | numeric(4,0) | | main | |
ename | character varying | | extended | |
job | character varying | | extended | |
arr_col | integer[] | | extended | |
Has OIDs: no
postgres=# select * from emp;
empno | ename | job | arr_col
---++-+
7900 | JAMES | CLERK | {1,2,7900}
7902 | FORD | ANALYST | {1,2,7902}
7934 | MILLER | CLERK | {1,2,7934}
(3 rows)
You will notice that the last element of the arr_col array is same as the empno of that
row.
The ECPG program tries to read the rows using FETCH in a structure emp defined
as
15 struct employee {
16 int empno;
17char ename[11];
18char job[15];
19int arr_col[3];
20 };
and then print the read contents as
39 /* Print members of the structure. */
40 for ( i = 0 ;i < 3; i++){
41 printf("empno=%d, ename=%s, job=%s, arr_col[2]=%d\n", emp[i].empno,
emp[i].ename, emp[i].job, emp[i].arr_col[2]);
42
43 }
But garbage values are printed
[ashutosh@ubuntu repro]./array_test
+++
empno=7900, ename=JAMES, job=CLERK, arr_col[2]=1
empno=2, ename=� , job=ANALYST, arr_col[2]=32767
empno=7934, ename=MILLER, job=CLERK, arr_col[2]=1719202336
Here are steps I have used to compile the ECPG program
[ashutosh@ubuntu repro]make array_test
ecpg -c -I/work/pg_head/build/include array_test.pgc
cc -I/work/pg_head/build/include -g -c -o array_test.o array_test.c
cc -g array_test.o -L/work/pg_head/build/lib -lecpg -lpq -o array_test
rm array_test.o array_test.c
where /work/pg_head/build is the directory containing the postgresql build (essentially
argument to the --prefix option to configure).
The programs compiles and links fine.
Without the arr_col member, the program works fine. So, it seems to be a problem with
array within structure array.
In array_test.c I see that the ECPGdo statement corresponding to the FETCH command is as
follows
87 /* Fetch multiple columns into one structure. */
88 { ECPGdo(__LINE__, 0, 1, NULL, 0, ECPGst_normal, "fetch 3 from cur1",
ECPGt_EOIT,
89 ECPGt_int,&(emp->empno),(long)1,(long)14,sizeof( struct employee ),
90 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
91 ECPGt_char,&(emp->ename),(long)11,(long)14,sizeof( struct employee ),
92 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
93 ECPGt_char,&(emp->job),(long)15,(long)14,sizeof( struct employee ),
94 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
95 ECPGt_int,(emp->arr_col),(long)1,(long)3,sizeof(int),
96 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EORT);
For all the members of struct employee, except arr_col, the size of array is set to 14
and next member offset is set of sizeof (struct employee). But for arr_col they are set
to 3 and sizeof(int) resp. So, for the next row onwards, the calculated offset of
arr_col member would not coincide with the real arr_col member's address.
Am I missing something here?
ECPG (I think intentionally) doesn't interpret or parse array fields.
You need to pass a character array and parse the contents by yourself.
Best regards,
Zoltán Böszörményi
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
