Re: [HACKERS] pl/python invalidate functions with composite arguments
On ons, 2011-02-09 at 10:09 +0100, Jan Urbański wrote: On 27/01/11 22:42, Jan Urbański wrote: On 23/12/10 14:50, Jan Urbański wrote: Here's a patch implementing properly invalidating functions that have composite type arguments after the type changes, as mentioned in http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's an incremental patch on top of the plpython-refactor patch sent eariler. Updated to master. Again. Committed. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python invalidate functions with composite arguments
On Thu, Feb 10, 2011 at 9:06 PM, Alex Hunsaker bada...@gmail.com wrote: On Wed, Feb 9, 2011 at 02:09, Jan Urbański wulc...@wulczer.org wrote: On 27/01/11 22:42, Jan Urbański wrote: On 23/12/10 14:50, Jan Urbański wrote: Here's a patch implementing properly invalidating functions that have composite type arguments after the type changes, as mentioned in http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's an incremental patch on top of the plpython-refactor patch sent eariler. Updated to master. Again. Looks good, it works as described, the code is clean and well documented and it passes the added regression tests. I took the liberty of looking at the other pls to see how they handled this to find they don't cache them in the first place. For fun, I hacked plpython to not cache to see if there was any performance difference pre patch, post patch and in the non-cached cases. I couldn't find any probably due to: 1) my simple test case (select count(test_composite_table_input('(John, 100, (10))')) FROM generate_series(1, 100);) 2) things being cached 3) cache invalidation having to do most of the work that the non caching cache does. I think there is one or two more SearchSysCall's. 4) overhead from cassert It seems a bit heavy handed to invalidate and remake the entire plpython function whenever we hit this case. I think we could get away with setting -is_rowtype = 2 in PLy_procedure_valid() instead. I suppose it should be fairly rare case anyway so... *shrug*. This last comment might make me think that we're looking for a new version of this patch, but the comment on the CommitFest application says looks good. So I'm not sure whether this should be marked Waiting on Author or Ready for Committer, but the current status of Needs Review looks wrong. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python invalidate functions with composite arguments
On 11/02/11 16:47, Robert Haas wrote: On Thu, Feb 10, 2011 at 9:06 PM, Alex Hunsaker bada...@gmail.com wrote: On Wed, Feb 9, 2011 at 02:09, Jan Urbański wulc...@wulczer.org wrote: It seems a bit heavy handed to invalidate and remake the entire plpython function whenever we hit this case. I think we could get away with setting -is_rowtype = 2 in PLy_procedure_valid() instead. I suppose it should be fairly rare case anyway so... *shrug*. This last comment might make me think that we're looking for a new version of this patch, but the comment on the CommitFest application says looks good. So I'm not sure whether this should be marked Waiting on Author or Ready for Committer, but the current status of Needs Review looks wrong. I'm not planning on writing a new version of the patch. AIUI Alex said, that there's a possible optimization to be done, but the gain is so small that it doesn't matter. Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python invalidate functions with composite arguments
On Fri, Feb 11, 2011 at 10:51 AM, Jan Urbański wulc...@wulczer.org wrote: On 11/02/11 16:47, Robert Haas wrote: On Thu, Feb 10, 2011 at 9:06 PM, Alex Hunsaker bada...@gmail.com wrote: On Wed, Feb 9, 2011 at 02:09, Jan Urbański wulc...@wulczer.org wrote: It seems a bit heavy handed to invalidate and remake the entire plpython function whenever we hit this case. I think we could get away with setting -is_rowtype = 2 in PLy_procedure_valid() instead. I suppose it should be fairly rare case anyway so... *shrug*. This last comment might make me think that we're looking for a new version of this patch, but the comment on the CommitFest application says looks good. So I'm not sure whether this should be marked Waiting on Author or Ready for Committer, but the current status of Needs Review looks wrong. I'm not planning on writing a new version of the patch. AIUI Alex said, that there's a possible optimization to be done, but the gain is so small that it doesn't matter. OK, marked Ready for Committer. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python invalidate functions with composite arguments
On Fri, Feb 11, 2011 at 08:47, Robert Haas robertmh...@gmail.com wrote: On Thu, Feb 10, 2011 at 9:06 PM, Alex Hunsaker bada...@gmail.com wrote: On Wed, Feb 9, 2011 at 02:09, Jan Urbański wulc...@wulczer.org wrote: On 27/01/11 22:42, Jan Urbański wrote: On 23/12/10 14:50, Jan Urbański wrote: Here's a patch implementing properly invalidating functions that have composite type arguments after the type changes, as mentioned in http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's an incremental patch on top of the plpython-refactor patch sent eariler. Updated to master. Again. Looks good, it works as described, the code is clean and well documented and it passes the added regression tests. I took the liberty of looking at the other pls to see how they handled this to find they don't cache them in the first place. For fun, I hacked plpython to not cache to see if there was any performance difference pre patch, post patch and in the non-cached cases. I couldn't find any probably due to: 1) my simple test case (select count(test_composite_table_input('(John, 100, (10))')) FROM generate_series(1, 100);) 2) things being cached 3) cache invalidation having to do most of the work that the non caching cache does. I think there is one or two more SearchSysCall's. 4) overhead from cassert It seems a bit heavy handed to invalidate and remake the entire plpython function whenever we hit this case. I think we could get away with setting -is_rowtype = 2 in PLy_procedure_valid() instead. I suppose it should be fairly rare case anyway so... *shrug*. This last comment might make me think that we're looking for a new version of this patch, but the comment on the CommitFest application says looks good. So I'm not sure whether this should be marked Waiting on Author or Ready for Committer, but the current status of Needs Review looks wrong. Drat, Ive been found it. I just didn't want to make things to easy for you. :) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python invalidate functions with composite arguments
On Wed, Feb 9, 2011 at 02:09, Jan Urbański wulc...@wulczer.org wrote: On 27/01/11 22:42, Jan Urbański wrote: On 23/12/10 14:50, Jan Urbański wrote: Here's a patch implementing properly invalidating functions that have composite type arguments after the type changes, as mentioned in http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's an incremental patch on top of the plpython-refactor patch sent eariler. Updated to master. Again. Looks good, it works as described, the code is clean and well documented and it passes the added regression tests. I took the liberty of looking at the other pls to see how they handled this to find they don't cache them in the first place. For fun, I hacked plpython to not cache to see if there was any performance difference pre patch, post patch and in the non-cached cases. I couldn't find any probably due to: 1) my simple test case (select count(test_composite_table_input('(John, 100, (10))')) FROM generate_series(1, 100);) 2) things being cached 3) cache invalidation having to do most of the work that the non caching cache does. I think there is one or two more SearchSysCall's. 4) overhead from cassert It seems a bit heavy handed to invalidate and remake the entire plpython function whenever we hit this case. I think we could get away with setting -is_rowtype = 2 in PLy_procedure_valid() instead. I suppose it should be fairly rare case anyway so... *shrug*. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python invalidate functions with composite arguments
On 27/01/11 22:42, Jan Urbański wrote: On 23/12/10 14:50, Jan Urbański wrote: Here's a patch implementing properly invalidating functions that have composite type arguments after the type changes, as mentioned in http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's an incremental patch on top of the plpython-refactor patch sent eariler. Updated to master. Again. diff --git a/src/pl/plpython/expected/plpython_types.out b/src/pl/plpython/expected/plpython_types.out index e74a400..d5f2c70 100644 *** a/src/pl/plpython/expected/plpython_types.out --- b/src/pl/plpython/expected/plpython_types.out *** SELECT * FROM test_type_conversion_array *** 603,608 --- 603,660 ERROR: return value of function with array return type is not a Python sequence CONTEXT: while creating return value PL/Python function test_type_conversion_array_error + --- + --- Composite types + --- + CREATE TABLE employee ( + name text, + basesalary integer, + bonus integer + ); + INSERT INTO employee VALUES ('John', 100, 10), ('Mary', 200, 10); + CREATE OR REPLACE FUNCTION test_composite_table_input(e employee) RETURNS integer AS $$ + return e['basesalary'] + e['bonus'] + $$ LANGUAGE plpythonu; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + ALTER TABLE employee DROP bonus; + SELECT name, test_composite_table_input(employee.*) FROM employee; + ERROR: KeyError: 'bonus' + CONTEXT: PL/Python function test_composite_table_input + ALTER TABLE employee ADD bonus integer; + UPDATE employee SET bonus = 10; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + CREATE TYPE named_pair AS ( + i integer, + j integer + ); + CREATE OR REPLACE FUNCTION test_composite_type_input(p named_pair) RETURNS integer AS $$ + return sum(p.values()) + $$ LANGUAGE plpythonu; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + + ALTER TYPE named_pair RENAME TO named_pair_2; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + -- -- Prepared statements -- diff --git a/src/pl/plpython/expected/plpython_types_3.out b/src/pl/plpython/expected/plpython_types_3.out index 577c1ff..ca81b08 100644 *** a/src/pl/plpython/expected/plpython_types_3.out --- b/src/pl/plpython/expected/plpython_types_3.out *** SELECT * FROM test_type_conversion_array *** 603,608 --- 603,660 ERROR: return value of function with array return type is not a Python sequence CONTEXT: while creating return value PL/Python function test_type_conversion_array_error + --- + --- Composite types + --- + CREATE TABLE employee ( + name text, + basesalary integer, + bonus integer + ); + INSERT INTO employee VALUES ('John', 100, 10), ('Mary', 200, 10); + CREATE OR REPLACE FUNCTION test_composite_table_input(e employee) RETURNS integer AS $$ + return e['basesalary'] + e['bonus'] + $$ LANGUAGE plpython3u; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + ALTER TABLE employee DROP bonus; + SELECT name, test_composite_table_input(employee.*) FROM employee; + ERROR: KeyError: 'bonus' + CONTEXT: PL/Python function test_composite_table_input + ALTER TABLE employee ADD bonus integer; + UPDATE employee SET bonus = 10; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + CREATE TYPE named_pair AS ( + i integer, + j integer + ); + CREATE OR REPLACE FUNCTION test_composite_type_input(p named_pair) RETURNS integer AS $$ + return sum(p.values()) + $$ LANGUAGE plpython3u; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + + ALTER TYPE named_pair RENAME TO named_pair_2; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + -- -- Prepared statements -- diff --git a/src/pl/plpython/plpython.c b/src/pl/plpython/plpython.c index fff7de7..3c0f35b 100644 *** a/src/pl/plpython/plpython.c --- b/src/pl/plpython/plpython.c *** typedef int Py_ssize_t; *** 101,106 --- 101,107 #include
Re: [HACKERS] pl/python invalidate functions with composite arguments
On 23/12/10 14:50, Jan Urbański wrote: Here's a patch implementing properly invalidating functions that have composite type arguments after the type changes, as mentioned in http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's an incremental patch on top of the plpython-refactor patch sent eariler. Updated to master. diff --git a/src/pl/plpython/expected/plpython_types.out b/src/pl/plpython/expected/plpython_types.out index 982005b..cf7b7de 100644 *** a/src/pl/plpython/expected/plpython_types.out --- b/src/pl/plpython/expected/plpython_types.out *** SELECT * FROM test_type_conversion_array *** 599,604 --- 599,656 ERROR: return value of function with array return type is not a Python sequence CONTEXT: while creating return value PL/Python function test_type_conversion_array_error + --- + --- Composite types + --- + CREATE TABLE employee ( + name text, + basesalary integer, + bonus integer + ); + INSERT INTO employee VALUES ('John', 100, 10), ('Mary', 200, 10); + CREATE OR REPLACE FUNCTION test_composite_table_input(e employee) RETURNS integer AS $$ + return e['basesalary'] + e['bonus'] + $$ LANGUAGE plpythonu; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + ALTER TABLE employee DROP bonus; + SELECT name, test_composite_table_input(employee.*) FROM employee; + ERROR: KeyError: 'bonus' + CONTEXT: PL/Python function test_composite_table_input + ALTER TABLE employee ADD bonus integer; + UPDATE employee SET bonus = 10; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + CREATE TYPE named_pair AS ( + i integer, + j integer + ); + CREATE OR REPLACE FUNCTION test_composite_type_input(p named_pair) RETURNS integer AS $$ + return sum(p.values()) + $$ LANGUAGE plpythonu; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + + ALTER TYPE named_pair RENAME TO named_pair_2; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + -- -- Prepared statements -- diff --git a/src/pl/plpython/expected/plpython_types_3.out b/src/pl/plpython/expected/plpython_types_3.out index eeb23b7..e10b060 100644 *** a/src/pl/plpython/expected/plpython_types_3.out --- b/src/pl/plpython/expected/plpython_types_3.out *** SELECT * FROM test_type_conversion_array *** 599,604 --- 599,656 ERROR: return value of function with array return type is not a Python sequence CONTEXT: while creating return value PL/Python function test_type_conversion_array_error + --- + --- Composite types + --- + CREATE TABLE employee ( + name text, + basesalary integer, + bonus integer + ); + INSERT INTO employee VALUES ('John', 100, 10), ('Mary', 200, 10); + CREATE OR REPLACE FUNCTION test_composite_table_input(e employee) RETURNS integer AS $$ + return e['basesalary'] + e['bonus'] + $$ LANGUAGE plpython3u; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + ALTER TABLE employee DROP bonus; + SELECT name, test_composite_table_input(employee.*) FROM employee; + ERROR: KeyError: 'bonus' + CONTEXT: PL/Python function test_composite_table_input + ALTER TABLE employee ADD bonus integer; + UPDATE employee SET bonus = 10; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + CREATE TYPE named_pair AS ( + i integer, + j integer + ); + CREATE OR REPLACE FUNCTION test_composite_type_input(p named_pair) RETURNS integer AS $$ + return sum(p.values()) + $$ LANGUAGE plpython3u; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + + ALTER TYPE named_pair RENAME TO named_pair_2; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + -- -- Prepared statements -- diff --git a/src/pl/plpython/plpython.c b/src/pl/plpython/plpython.c index aafe556..54605fc 100644 *** a/src/pl/plpython/plpython.c --- b/src/pl/plpython/plpython.c *** typedef int Py_ssize_t; *** 101,106 --- 101,107 #include nodes/makefuncs.h #include parser/parse_type.h
[HACKERS] pl/python invalidate functions with composite arguments
Here's a patch implementing properly invalidating functions that have composite type arguments after the type changes, as mentioned in http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's an incremental patch on top of the plpython-refactor patch sent eariler. Git branch for this patch: https://github.com/wulczer/postgres/tree/invalidate-composites. The idea in general is for this to work (taken straight from the unit tests, btw) CREATE TABLE employee ( name text, basesalary integer, bonus integer ); INSERT INTO employee VALUES ('John', 100, 10), ('Mary', 200, 10); CREATE OR REPLACE FUNCTION test_composite_table_input(e employee) RETURNS integer AS $$ return e['basesalary'] + e['bonus'] $$ LANGUAGE plpythonu; SELECT name, test_composite_table_input(employee.*) FROM employee; ALTER TABLE employee DROP bonus; -- this fails SELECT name, test_composite_table_input(employee.*) FROM employee; ALTER TABLE employee ADD bonus integer; UPDATE employee SET bonus = 10; -- this works again SELECT name, test_composite_table_input(employee.*) FROM employee; It's a long-standing TODO item, and a generally good thing to do. Cheers, Jan diff --git a/src/pl/plpython/expected/plpython_types.out b/src/pl/plpython/expected/plpython_types.out index 982005b..cf7b7de 100644 *** a/src/pl/plpython/expected/plpython_types.out --- b/src/pl/plpython/expected/plpython_types.out *** SELECT * FROM test_type_conversion_array *** 599,604 --- 599,656 ERROR: return value of function with array return type is not a Python sequence CONTEXT: while creating return value PL/Python function test_type_conversion_array_error + --- + --- Composite types + --- + CREATE TABLE employee ( + name text, + basesalary integer, + bonus integer + ); + INSERT INTO employee VALUES ('John', 100, 10), ('Mary', 200, 10); + CREATE OR REPLACE FUNCTION test_composite_table_input(e employee) RETURNS integer AS $$ + return e['basesalary'] + e['bonus'] + $$ LANGUAGE plpythonu; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + ALTER TABLE employee DROP bonus; + SELECT name, test_composite_table_input(employee.*) FROM employee; + ERROR: KeyError: 'bonus' + CONTEXT: PL/Python function test_composite_table_input + ALTER TABLE employee ADD bonus integer; + UPDATE employee SET bonus = 10; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + CREATE TYPE named_pair AS ( + i integer, + j integer + ); + CREATE OR REPLACE FUNCTION test_composite_type_input(p named_pair) RETURNS integer AS $$ + return sum(p.values()) + $$ LANGUAGE plpythonu; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + + ALTER TYPE named_pair RENAME TO named_pair_2; + SELECT test_composite_type_input(row(1, 2)); + test_composite_type_input + --- + 3 + (1 row) + -- -- Prepared statements -- diff --git a/src/pl/plpython/expected/plpython_types_3.out b/src/pl/plpython/expected/plpython_types_3.out index eeb23b7..e10b060 100644 *** a/src/pl/plpython/expected/plpython_types_3.out --- b/src/pl/plpython/expected/plpython_types_3.out *** SELECT * FROM test_type_conversion_array *** 599,604 --- 599,656 ERROR: return value of function with array return type is not a Python sequence CONTEXT: while creating return value PL/Python function test_type_conversion_array_error + --- + --- Composite types + --- + CREATE TABLE employee ( + name text, + basesalary integer, + bonus integer + ); + INSERT INTO employee VALUES ('John', 100, 10), ('Mary', 200, 10); + CREATE OR REPLACE FUNCTION test_composite_table_input(e employee) RETURNS integer AS $$ + return e['basesalary'] + e['bonus'] + $$ LANGUAGE plpython3u; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + ALTER TABLE employee DROP bonus; + SELECT name, test_composite_table_input(employee.*) FROM employee; + ERROR: KeyError: 'bonus' + CONTEXT: PL/Python function test_composite_table_input + ALTER TABLE employee ADD bonus integer; + UPDATE employee SET bonus = 10; + SELECT name, test_composite_table_input(employee.*) FROM employee; + name | test_composite_table_input + --+ + John |110 + Mary |210 + (2 rows) + + CREATE TYPE named_pair AS ( + i integer, + j