Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS

2015-05-08 Thread Tom Lane
Christian Ullrich ch...@chrullrich.net writes:
 * Peter Eisentraut wrote:
 On 4/30/15 2:49 PM, Andrew Dunstan wrote:
 friarbird is a FreeBSD buildfarm animal running with
 -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours.
 However, it's been stuck since Monday running the plpython regression
 tests. The only relevant commit seems to be the transforms feature.

 I can reproduce it.  I'll look into it.

 I looked over the CCA animals and noticed that pademelon and gaur are 
 apparently unaffected;

pademelon and gaur do not run CCA (if they did, it would take weeks for
a run to complete :-().

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS

2015-05-08 Thread Christian Ullrich

* Peter Eisentraut wrote:


On 4/30/15 2:49 PM, Andrew Dunstan wrote:



friarbird is a FreeBSD buildfarm animal running with
-DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours.
However, it's been stuck since Monday running the plpython regression
tests. The only relevant commit seems to be the transforms feature.


I can reproduce it.  I'll look into it.


I looked over the CCA animals and noticed that pademelon and gaur are 
apparently unaffected; what they have in common is the OS (HP-UX) and 
the Python version (2.5). There's nothing I can do about OS-related 
differences, but I thought I'd check the Python angle.


With Python 2.5.6, jaguarundi locks up on the plpython tests just the 
same as with 3.4, and friarbird with 2.7. So that is not the culprit, 
either.


I ran make check by hand, and noticed three tests where it seemed to 
hang (I gave it at least three minutes each, and the functions in the 
queries are simple):


plpython_spiSELECT cursor_plan();
plpython_setof  SELECT test_setof_as_list(1, 'list');
plpython_composite  SELECT multiout_simple_setof();

These are the only plpython tests that mention SETOF at all, and the 
queries that hung are the first ones in their respective tests to 
actually build a set.


Does that help?

--
Christian




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS

2015-05-08 Thread Christian Ullrich

* Tom Lane wrote:


Christian Ullrich ch...@chrullrich.net writes:

* Peter Eisentraut wrote:

On 4/30/15 2:49 PM, Andrew Dunstan wrote:

friarbird is a FreeBSD buildfarm animal running with
-DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours.
However, it's been stuck since Monday running the plpython regression
tests. The only relevant commit seems to be the transforms feature.



I can reproduce it.  I'll look into it.



I looked over the CCA animals and noticed that pademelon and gaur are
apparently unaffected;


pademelon and gaur do not run CCA (if they did, it would take weeks for
a run to complete :-().


Ah. I obviously had associated the note icon with CCA. Sorry. OTOH, if 
I had noticed that, I might not have gone into more detail ...


--
Christian





--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Transforms patch not respecting if indentation

2015-05-07 Thread Magnus Hagander
On Mon, May 4, 2015 at 6:45 AM, Michael Paquier michael.paqu...@gmail.com
wrote:

 Hi all,

 Coverity complained about a small indentation issue in ruleutils.c:
 +   appendStringInfoString(buf, \n TRANSFORM );
 +   for (i = 0; i  ntypes; i++)
 +   {
 +   if (i != 0)
 +   appendStringInfoString(buf, , );
 +   appendStringInfo(buf, FOR TYPE %s,
 format_type_be(trftypes[i]));
 +   }


Applied, thanks.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS

2015-05-01 Thread Andrew Dunstan


On 05/01/2015 08:57 AM, Andrew Dunstan wrote:


On 04/30/2015 09:09 PM, Christian Ullrich wrote:

* Andrew Dunstan:


friarbird is a FreeBSD buildfarm animal running with
-DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours.
However, it's been stuck since Monday running the plpython regression
tests. The only relevant commit seems to be the transforms feature.
Here's what it's been doing:



query| SELECT cursor_plan();


Same here, on jaguarundi. I actually killed it intentionally this 
morning, hoping that whatever the problem was might have been fixed 
already. No such luck.


I would suspect that it might have something to do with the OS, if 
all the other CCA animals weren't lining up nicely behind in the 
buildfarm status page.






Yeah, this happened again this morning, so it seems to be quite 
reliably reproducible. I killed it and I've set friarbird to build 
without python for now, but this is clearly an issue that needs to be 
resolved.



And I have confirmed that it's not an OS problem - I have reproduced it 
on a modern Linux instance (it's still running, in fact). So it's quite 
clearly a bug that needs to be fixed.


cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS

2015-05-01 Thread Andrew Dunstan


On 04/30/2015 09:09 PM, Christian Ullrich wrote:

* Andrew Dunstan:


friarbird is a FreeBSD buildfarm animal running with
-DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours.
However, it's been stuck since Monday running the plpython regression
tests. The only relevant commit seems to be the transforms feature.
Here's what it's been doing:



query| SELECT cursor_plan();


Same here, on jaguarundi. I actually killed it intentionally this 
morning, hoping that whatever the problem was might have been fixed 
already. No such luck.


I would suspect that it might have something to do with the OS, if all 
the other CCA animals weren't lining up nicely behind in the buildfarm 
status page.



I imagine it was in some sort of infinite loop. gdb says it's all in
src/backend/utils/cache/plancache.c, although not the same line each
time I run it.


I ktrace'd it this morning, but cleverly did not keep the dump. It 
looked much the same to me, though, it was reading the same filenode 
over and over again.





Yeah, this happened again this morning, so it seems to be quite reliably 
reproducible. I killed it and I've set friarbird to build without python 
for now, but this is clearly an issue that needs to be resolved.


Side thought - maybe we need some sort of timeout mechanism for the 
buildfarm to try to stop it from hanging. There is actually some timeout 
code in there from back in the CVS days when occasionally CVS would 
hang. It could be adapted to timeout other steps.


cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS

2015-04-30 Thread Christian Ullrich

* Andrew Dunstan:


friarbird is a FreeBSD buildfarm animal running with
-DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours.
However, it's been stuck since Monday running the plpython regression
tests. The only relevant commit seems to be the transforms feature.
Here's what it's been doing:



query| SELECT cursor_plan();


Same here, on jaguarundi. I actually killed it intentionally this 
morning, hoping that whatever the problem was might have been fixed 
already. No such luck.


I would suspect that it might have something to do with the OS, if all 
the other CCA animals weren't lining up nicely behind in the buildfarm 
status page.



I imagine it was in some sort of infinite loop. gdb says it's all in
src/backend/utils/cache/plancache.c, although not the same line each
time I run it.


I ktrace'd it this morning, but cleverly did not keep the dump. It 
looked much the same to me, though, it was reading the same filenode 
over and over again.


--
Christian




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms vs. CLOBBER_CACHE_ALWAYS

2015-04-30 Thread Peter Eisentraut
On 4/30/15 2:49 PM, Andrew Dunstan wrote:
 friarbird is a FreeBSD buildfarm animal running with
 -DCLOBBER_CACHE_ALWAYS. It usually completes a run in about 6.5 hours.
 However, it's been stuck since Monday running the plpython regression
 tests. The only relevant commit seems to be the transforms feature.

I can reproduce it.  I'll look into it.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-06-12 Thread Peter Eisentraut
On 6/12/13 1:20 AM, Josh Berkus wrote:
 Peter, All:
 
 Does anyone feel like fixing the LOAD issue with transforms?  I haven't
 seen any activity on the patch.

I plan to send in an updated patch.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-06-11 Thread Josh Berkus
Peter, All:

Does anyone feel like fixing the LOAD issue with transforms?  I haven't
seen any activity on the patch.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-18 Thread Josh Berkus
On 03/13/2013 09:54 AM, Dimitri Fontaine wrote:
 Peter Eisentraut pete...@gmx.net writes:
 At run time, this will sort itself out, because all the required modules
 will be loaded.  The problem is when you create the glue extension and
 haven't invoked any code from any of the dependent extension in the
 session.  Abstractly, the possible solutions are either not to check the
 functions when the extension is created (possibly settable by a flag) or
 to somehow force a load of all dependent extensions when the new
 extension is created.  (I say extension here even though dynamically
 loadable modules are attached to functions, which makes this even more
 confusing.)
 
 I think CREATE EXTENSION should be LOADing all distinct modules
 referenced from all required extensions functions. It does sound easy
 enough to code, and I can't imagine why we would want to not do that.

Where are we with this?  Will the loading issue be fixed, or should we
bump this feature to 9.4?


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-13 Thread Dimitri Fontaine
Peter Eisentraut pete...@gmx.net writes:
 At run time, this will sort itself out, because all the required modules
 will be loaded.  The problem is when you create the glue extension and
 haven't invoked any code from any of the dependent extension in the
 session.  Abstractly, the possible solutions are either not to check the
 functions when the extension is created (possibly settable by a flag) or
 to somehow force a load of all dependent extensions when the new
 extension is created.  (I say extension here even though dynamically
 loadable modules are attached to functions, which makes this even more
 confusing.)

I think CREATE EXTENSION should be LOADing all distinct modules
referenced from all required extensions functions. It does sound easy
enough to code, and I can't imagine why we would want to not do that.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-12 Thread Andres Freund
On 2013-03-11 20:28:05 -0400, Peter Eisentraut wrote:
 On Mon, 2013-03-11 at 18:11 +0100, Andres Freund wrote:
  If we don't find a better solution, yes. Why don't we lookup type
  input/ouput function for parameters and return type during CREATE
  FUNCTION? That should solve the issue in a neater way?

 A function in general has no particular use for a type's input or output
 function.

 Also, a type's input/output functions are not necessarily in the same
 library as other things about that type that you might want or need.

In theory they are not necessarily tied together, yes. In practice they
nearly always are in the same shared object.
The lookup for them afaics are the only reason why e.g. the
plperl-hstore transform works outside of CREATE EXTENSION without
explictly doing something ugly like:

On 03/11/2013 09:50 AM, Andres Freund wrote:
 DO LANGUAGE plperlu ;
 SELECT NULL::hstore;

beforehand.

If you have a better idea than putting the above into the extension
script or making a function look up its parameters, I am all ears. It
sure would only be a hack.

Greetings,

Andres Freund

--
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-11 Thread Josh Berkus

 Your error looks like youre erroring out in plperl not in hstore?

Look again.

Peter, is there any way for you to tackle this issue?  I have no idea
how to fix it, myself ...

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-11 Thread Andres Freund
On 2013-03-11 09:42:18 -0700, Josh Berkus wrote:
 
  Your error looks like youre erroring out in plperl not in hstore?
 
 Look again.

 ERROR:  could not load library 
 /home/josh/pg93/lib/postgresql/hstore_plperl.so:
 /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key

Thats a perl symbol.

 Peter, is there any way for you to tackle this issue?  I have no idea
 how to fix it, myself ...

If you add a:

DO LANGUAGE plperlu ;
SELECT NULL::hstore;

to the extension script, does it work?

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-11 Thread Josh Berkus
On 03/11/2013 09:50 AM, Andres Freund wrote:
 DO LANGUAGE plperlu ;
 SELECT NULL::hstore;

Aha, so that's what you meant!

Yeah, it works if I reference both extensions before the CREATE EXTENSION.

In that case, seems like that could be added to the extension SQL, no?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-11 Thread Andres Freund
On 2013-03-11 09:55:34 -0700, Josh Berkus wrote:
 On 03/11/2013 09:50 AM, Andres Freund wrote:
  DO LANGUAGE plperlu ;
  SELECT NULL::hstore;
 
 Aha, so that's what you meant!
 
 Yeah, it works if I reference both extensions before the CREATE EXTENSION.
 
 In that case, seems like that could be added to the extension SQL, no?

If we don't find a better solution, yes. Why don't we lookup type
input/ouput function for parameters and return type during CREATE
FUNCTION? That should solve the issue in a neater way?

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-11 Thread Peter Eisentraut
On Mon, 2013-03-11 at 18:11 +0100, Andres Freund wrote:
 If we don't find a better solution, yes. Why don't we lookup type
 input/ouput function for parameters and return type during CREATE
 FUNCTION? That should solve the issue in a neater way? 

A function in general has no particular use for a type's input or output
function.

Also, a type's input/output functions are not necessarily in the same
library as other things about that type that you might want or need.




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-06 Thread Peter Eisentraut
On 3/5/13 5:52 PM, Josh Berkus wrote:
 More on this: the problem appears to be that the symbols for hstore are
 loaded only if I've just just created the extension in that database:

I see.  This is a problem with any kind of dynamically loadable module
that uses another module.  The other module is only loaded either at
creation time or when a function from it is first used (or if a preload
mechanism is used).

At run time, this will sort itself out, because all the required modules
will be loaded.  The problem is when you create the glue extension and
haven't invoked any code from any of the dependent extension in the
session.  Abstractly, the possible solutions are either not to check the
functions when the extension is created (possibly settable by a flag) or
to somehow force a load of all dependent extensions when the new
extension is created.  (I say extension here even though dynamically
loadable modules are attached to functions, which makes this even more
confusing.)

In normal programming languages, this is normally addressed by placing
explicit load/require/import statements before you do anything else.
What we are doing here is more like an autoload functionality that some
environments have.  Those have the same problem.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-06 Thread Josh Berkus
Peter,

 At run time, this will sort itself out, because all the required modules
 will be loaded.  The problem is when you create the glue extension and
 haven't invoked any code from any of the dependent extension in the
 session.  

Just invoking code doesn't seem to be enough.  I tried just using the
Hstore data type, and then loading hstore_plperl, but that still didn't
work.  It seems like only CREATE EXTENSION loads *all* the symbols.

 Abstractly, the possible solutions are either not to check the
 functions when the extension is created (possibly settable by a flag) or
 to somehow force a load of all dependent extensions when the new
 extension is created. 

The latter would be ideal, but I don't know that we currently have any
mechanism for it.

--Josh

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-06 Thread Andres Freund
On 2013-03-06 09:53:29 -0800, Josh Berkus wrote:
 Peter,
 
  At run time, this will sort itself out, because all the required modules
  will be loaded.  The problem is when you create the glue extension and
  haven't invoked any code from any of the dependent extension in the
  session.  
 
 Just invoking code doesn't seem to be enough.  I tried just using the
 Hstore data type, and then loading hstore_plperl, but that still didn't
 work.  It seems like only CREATE EXTENSION loads *all* the symbols.

Your error looks like youre erroring out in plperl not in hstore?

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Steve Singer

On 13-03-03 08:13 PM, Josh Berkus wrote:

This (creating the extensions) works fine for me on a  Ubuntu 10.x system




Now if only we can work out the combinatorics issue ...



The plpython2u extensions worked fine but I haven't been able to get 
this to work with plpython3u (python 3.1).


create extension hstore_plpython3u;
ERROR:  could not load library 
/usr/local/pgsql93git/lib/hstore_plpython3.so: 
/usr/local/pgsql93git/lib/hstore_plpython3.so: undefined symbol: 
_Py_NoneStruct



Peter mentioned in the submission that the unit tests don't pass with 
python3, it doesn't  work for meoutside the tests either.


Steve




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Josh Berkus

 Peter mentioned in the submission that the unit tests don't pass with
 python3, it doesn't  work for meoutside the tests either.

Also, note that the feature is the concept of Transforms, not the
individual transforms which Peter provides.  The idea is to enable a new
kind of extension.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Peter Eisentraut
On 3/3/13 6:14 PM, Josh Berkus wrote:
 Currently Peter is punting (as is proper in a new patch) by having a
 separate extension for each combination (hstore/plperl, hstore/plpython,
 ltree/plpython, etc.).  This is obviously not a maintainable approach in
 the long run.

There is also a {Perl, Python, Ruby, etc.} binding for {libpq, libmysql,
libpng, libyaml, libssl, libgmp, etc.}, each as a separately
downloadable and buildable package.  I don't think anyone has ever
seriously considered a system by which if, say, you have Python and
libyaml installed, pyyaml magically appears.  Might be nice, but maybe
not.  The solution, in practice, is that you download pyyaml, and it
pulls in any required dependencies.  This would work the same way.




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Josh Berkus
Peter,

 There is also a {Perl, Python, Ruby, etc.} binding for {libpq, libmysql,
 libpng, libyaml, libssl, libgmp, etc.}, each as a separately
 downloadable and buildable package.  I don't think anyone has ever
 seriously considered a system by which if, say, you have Python and
 libyaml installed, pyyaml magically appears.  Might be nice, but maybe
 not.  The solution, in practice, is that you download pyyaml, and it
 pulls in any required dependencies.  This would work the same way.

That would be good too, although we don't currently have that
capability; if I try to install hstore_plpython without plpython
installed, it just errors out.  Aside from that, what can we reasonably
do for 9.3 to get this feature in?

Maybe we add a transforms/ subdirectory of contrib, so that it can be as
crowded as we want?  Or put the transforms on PGXN for now?

I want to see this feature go in so that the community starts writing
transforms this year instead of next year.

BTW, dependancies seem to be working OK for DROP:

postgres=# drop extension plpythonu;
ERROR:  cannot drop extension plpythonu because other objects depend on it
DETAIL:  extension hstore_plpythonu depends on extension plpythonu
function look_up_hstore(hstore,text) depends on transform for hstore
language plpythonu
extension ltree_plpythonu depends on extension plpythonu
HINT:  Use DROP ... CASCADE to drop the dependent objects too.
STATEMENT:  drop extension plpythonu;
ERROR:  cannot drop extension plpythonu because other objects depend on it
DETAIL:  extension hstore_plpythonu depends on extension plpythonu
function look_up_hstore(hstore,text) depends on transform for hstore
language plpythonu
extension ltree_plpythonu depends on extension plpythonu
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Josh Berkus
Peter,

I'm still getting intermittent linking failures:

postgres=# drop extension plperl cascade;
NOTICE:  drop cascades to extension hstore_plperl
DROP EXTENSION

postgres=# create extension plperl;
CREATE EXTENSION
postgres=# create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
hstoreUniquePairs
STATEMENT:  create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
hstoreUniquePairs
postgres=#

There appears to be something wonky which breaks when I've been running
9.2, shut it down, and fire up 9.3.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Josh Berkus

 postgres=# create extension plperl;
 CREATE EXTENSION
 postgres=# create extension hstore_plperl;
 ERROR:  could not load library
 /home/josh/pg93/lib/postgresql/hstore_plperl.so:
 /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
 hstoreUniquePairs
 STATEMENT:  create extension hstore_plperl;
 ERROR:  could not load library
 /home/josh/pg93/lib/postgresql/hstore_plperl.so:
 /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
 hstoreUniquePairs
 postgres=#
 
 There appears to be something wonky which breaks when I've been running
 9.2, shut it down, and fire up 9.3.

More on this: the problem appears to be that the symbols for hstore are
loaded only if I've just just created the extension in that database:

postgres=# create database plperlh
postgres-# ;
CREATE DATABASE
postgres=# \c plperlh;
You are now connected to database plperlh as user josh.
plperlh=# create extension plperl;
CREATE EXTENSION
plperlh=# create extension hstore;
CREATE EXTENSION
plperlh=# create extension hstore_plperl;
CREATE EXTENSION
plperlh=#

plperlh=# \c postgres
You are now connected to database postgres as user josh.
postgres=# create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
PL_thr_key
STATEMENT:  create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
PL_thr_key





-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Joshua D. Drake


On 03/05/2013 02:52 PM, Josh Berkus wrote:


plperlh=# \c postgres
You are now connected to database postgres as user josh.
postgres=# create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
PL_thr_key
STATEMENT:  create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
PL_thr_key


What happens if you set your LD_LIBRARY_PATH to reflect each 
installation before you start the database?


JD











--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-05 Thread Josh Berkus

 What happens if you set your LD_LIBRARY_PATH to reflect each
 installation before you start the database?

No change, at least not setting it to $PGHOME/lib.


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-03 Thread Josh Berkus
Peter,

I tried this patch out.  I didn't get as far as testing the
functionality because of errors.

configure/make/make install/make check worked, without asserts.  I
believe DF found some errors when he enabled assertions.

When I tried to install the actual transform extensions, though, it blew
up with symbol errors:

transforms=# create extension hstore
transforms-# ;
CREATE EXTENSION

transforms=# create extension ltree
transforms-# ;
CREATE EXTENSION

transforms=# create extension plperl
transforms-# ;
CREATE EXTENSION

transforms=# create extension plpythonu
transforms-# ;
CREATE EXTENSION

transforms=# create extension ltree_plpythonu;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/ltree_plpython2.so:
/home/josh/pg93/lib/postgresql/ltree_plpython2.so: undefined symbol:
PyList_New
STATEMENT:  create extension ltree_plpythonu;

transforms=# create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
hstoreUniquePairs
STATEMENT:  create extension hstore_plperl;

This surprised me, because make check for the extensions passed fine.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-03 Thread Josh Berkus
Peter, all:

So in addition to the bugs I encountered in getting this patch to work,
we have a design issue to work out: how to load all of the transform
functions.  Each transform depends on an optional datatype (like hstore)
and an optional external language (like plperl), which can be loaded
into the database in any order.

Currently Peter is punting (as is proper in a new patch) by having a
separate extension for each combination (hstore/plperl, hstore/plpython,
ltree/plpython, etc.).  This is obviously not a maintainable approach in
the long run.

In an ideal world, transformations would load automatically as soon as
all of their prerequisites load.  For example, if you load hstore and
plpythonU, then hstore_plpython should automatically load whenever the
second extension is loaded.

However, we currently have no mechanism for this, especially since we
don't know what order the user will load the two extensions in.
Realistically, this would require enhancing the extension mechanism to
track transformations.

Discuss?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-03 Thread Josh Berkus

 transforms=# create extension hstore_plperl;
 ERROR:  could not load library
 /home/josh/pg93/lib/postgresql/hstore_plperl.so:
 /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
 hstoreUniquePairs
 STATEMENT:  create extension hstore_plperl;
 
 This surprised me, because make check for the extensions passed fine.

Oh, this is on Ubuntu 12.10, not OSX.   So possibly the fixes you made
to fix linking on OSX broke other platforms.


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-03 Thread Steve Singer

On 13-03-03 06:15 PM, Josh Berkus wrote:

transforms=# create extension hstore_plperl;
ERROR:  could not load library
/home/josh/pg93/lib/postgresql/hstore_plperl.so:
/home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol:
hstoreUniquePairs
STATEMENT:  create extension hstore_plperl;

This surprised me, because make check for the extensions passed fine.

Oh, this is on Ubuntu 12.10, not OSX.   So possibly the fixes you made
to fix linking on OSX broke other platforms.




This (creating the extensions) works fine for me on a  Ubuntu 10.x system

template1=# create database test;
CREATE DATABASE
template1=# \c test
You are now connected to database test as user ssinger.
test=# create extension hstore;
CREATE EXTENSION
test=# create extension hstore_plpythonu;
ERROR:  required extension plpythonu is not installed
STATEMENT:  create extension hstore_plpythonu;
ERROR:  required extension plpythonu is not installed
test=# create extension plpythonu;
CREATE EXTENSION
test=# create extension hstore_plpythonu;
CREATE EXTENSION
test=#
test=# create extension plperl;
CREATE EXTENSION
test=# create extension hstore_plperl;
CREATE EXTENSION
test=# create extension plperlu;
CREATE EXTENSION
test=# create extension hstore_plperlu;
CREATE EXTENSION
test=#




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-03-03 Thread Josh Berkus

 This (creating the extensions) works fine for me on a  Ubuntu 10.x system
 

H.  So I wiped everything out and retried this with clean
directories, and it worked fine.  Now I'm not sure how I got the error
in the first place.

Anyway, the feature seems to work as expected:

create function look_up_hstore (
some_hs hstore, which_key text )
returns text
language plpythonu as $f$
return some_hs[which_key]
$f$;

postgres=# create table storit ( it hstore );
CREATE TABLE


postgres=# insert into storit values ( 'f = josh,l=berkus'
),('f=jan,l=wieck'),('f=tom,l=lane');
INSERT 0 3
postgres=# select look_up_hstore(it,'f') from storit;
 look_up_hstore

 josh
 jan
 tom
(3 rows)

I'll have to think of other ways to test it out, but right now it seems
to pass muster.  I was a little unclear on what Python data structure an
ltree should produce.

Now if only we can work out the combinatorics issue ...

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2013-01-15 Thread Peter Eisentraut
Here is an updated patch for the transforms feature.  The previous
discussion was here:
http://www.postgresql.org/message-id/1339713732.11971.79.ca...@vanquo.pezone.net

Old news: At the surface, this contains:

- New catalog pg_transform

- CREATE/DROP TRANSFORM

As proofs of concept and useful applications, I have included transforms for

- PL/Perl - hstore
- PL/Python - hstore
- PL/Python - ltree

New news: I have tried to address all issues raised during previous
reviews.

The regression tests for hstore_plpython and ltree_plpython don't pass
with Python 3.  This needs to be fixed, obviously, but it's an issue
unrelated to the core functionality.

As a demo that this can be used externally as well, I have written a
proof-of-concept transform between plpython and the mpz type in the pgmp
extension (multi-precision arithmetic extension):
https://github.com/petere/pgmp/tree/transforms/plpython

Note: There was a lot of churn lately in src/backend/commands/, and I
had to fix merge conflicts several times, so you need source as of about
right now to apply this patch cleanly.

Side issue/peculiarity: I had to change some shared library linking
flags for OS X.  Ordinarily, when linking a dynamically loadable module
on OS X, if there are unresolved symbols, it's an error.  But for
example, when we link hstore_plpython, we expect some symbols to be
resolved in hstore.so or plpythonu.so, so we need to ignore unresolved
symbols.  This change basically just makes OS X behave like other
platforms in this regard.  There might be other portability issues on
other platforms.



pg-transforms-20130115.patch.gz
Description: GNU Zip compressed data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2012-07-06 Thread Peter Eisentraut
I haven't had the time to finish all the issues with this, but I want to
keep the discussion going in the meantime and provide an updated patch.

On mån, 2012-06-18 at 17:33 +0200, Andres Freund wrote:

 Cursory code review:
 * pstrndup already exists as pnstrdup (hstore_plperl.c)

Fixed.

 * PyString_FromStringAndSize return value not decreffed? PyDict_SetItem 
 doesn't steal references

Fixed.

 * In plpython_to_hstore I would move the 'pairs' and some related variables 
 in 
 the PG_TRY block, so the reader doesn't need to check whether it should be 
 volatile
 * In the same function items needs to be volatile to fit into longjmp 
 semantics

I'll recheck that later.

 * I don't think recording dependencies on transforms used when creating 
 functions is a good idea as the transform might get created after the 
 functions already exists. That seems to be a pretty confusing behaviour.

We need the dependencies, because otherwise dropping a transform would
break or silently alter the behavior of functions that depend on it.
That sounds like my worst nightmare, thinking of some applications that
would be affected by that.  But your point is a good one.  I think this
could be addressed by prohibiting the creation of a transform that
affects functions that already exist.

Because the legacy behavior of PL implementations of defaulting to a
string representation conversion, we would technically need a dependency
on the absence of a transform object to make this airtight.  In the far
future, I could imagine removing this default behavior, meaning you
couldn't create the function if no suitable transforms exist for all
argument and return types.

 * I forsee the need for multiple transforms for the same type/language pair 
 to 
 coexist. The user would need to manually choose/call the transform in 
 that 
 case. This currently isn't easily possible...

I thought about this briefly at the beginning, but see under worst
nightmare above.  Also, having a configuration setting for this or
something would prevent any PL functions from being immutable.  We don't
allow multiple casts or multiple in/out functions either, which are
related concepts.  If you want different behavior, you should define a
different type or different language.

  *) No psql backslash commands yet.  Could be added.
 Doesn't really seem necessary to me. Not many people will need to look at 
 this 
 and the list of commands already is rather long.

I'm going to leave this out for now.

  *) Permissions: Transforms don't have owners, a bit like casts.
  Currently, you are allowed to drop a transform if you own both the type
  and the language.  That might be too strict, maybe own the type and have
  privileges on the language would be enough.
 Seems sensible enough to me.

I have made this change.

  *) There is currently some syntax schizophrenia.  The grammar accepts
  
  CREATE TRANSFORM FOR hstore LANGUAGE plperl (
  FROM SQL WITH hstore_to_plperl,
  TO SQL WITH plperl_to_hstore
  );
  
  but pg_dump produces
  
  CREATE TRANSFORM FOR hstore LANGUAGE plperl (
  FROM SQL WITH hstore_to_plperl(hstore),
  TO SQL WITH plperl_to_hstore(internal)
  );
  
  The SQL standard allows both.  (In the same way that it allows 'DROP
  FUNCTION foo' without arguments, if it is not ambigious.)  Precedent is
  that CREATE CAST requires arguments, but CREATE LANGUAGE does not.
 I don't find that problematic personally.

I have fixed the syntax to include argument types, so the dump output
and the input grammar is consistent.


Other changes:

- Fixed ecpg grammar to work again with this.

- Changed extension naming to be more consistent.

- Build additional contrib modules conditionally depending on whether
--with-perl or --with-python were configured.  (complaint from Jeff
Janes)

- Fixed Python 3.


Things I still want to do:

- Refactor the regression test framework for Python 3 so that contrib
modules or external extensions don't have to repeat the magic in
src/pl/plpython/Makefile.  (Python 3 with hstore_plpython and
ltree_plpython works, but the tests don't run.)

- Refactor pyobject_to_string(), which is currently kind of copied and
pasted from plpython, but should instead be exported by plpython in some
suitable way.

- Refactor shared library building so that I can have, say, hstore,
hstore_plperl, and hstore_plpython in one directory, rather than in
three.  The reason being, if someone has a new type in a repository on
github or something, I don't want them to have to make three separate
projects or some crazy subdirectory structure in order to add some PL
support for their type.  This will require some deep Makefile.shlib
hacking, but I think it's worth trying to make this simple(r).


So, it's quite likely that this patch won't get finished in this commit
fest.



transforms-20120707.patch.gz
Description: GNU Zip compressed data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your 

Re: [HACKERS] transforms

2012-06-18 Thread Andres Freund
Hi Peter,

On Friday, June 15, 2012 12:42:12 AM Peter Eisentraut wrote:
 Here is my first patch for the transforms feature.  This is a mechanism
 to adapt data types to procedural languages.  The previous proposal was
 here: http://archives.postgresql.org/pgsql-hackers/2012-05/msg00728.php
 
 At the surface, this contains:
 
 - New catalog pg_transform
 
 - CREATE/DROP TRANSFORM
Cursory code review:
* pstrndup already exists as pnstrdup (hstore_plperl.c)
* PyString_FromStringAndSize return value not decreffed? PyDict_SetItem 
doesn't steal references
* In plpython_to_hstore I would move the 'pairs' and some related variables in 
the PG_TRY block, so the reader doesn't need to check whether it should be 
volatile
* In the same function items needs to be volatile to fit into longjmp 
semantics
* I don't think recording dependencies on transforms used when creating 
functions is a good idea as the transform might get created after the 
functions already exists. That seems to be a pretty confusing behaviour.
* I am a bit wary that some place can be used to call functions accepting 
INTERNAL indirectly, couldn't think of any immediately though. Will look  into 
this a bit, but I am not experienced in the area, so ...
* I forsee the need for multiple transforms for the same type/language pair to 
coexist. The user would need to manually choose/call the transform in that 
case. This currently isn't easily possible...


 *) No psql backslash commands yet.  Could be added.
Doesn't really seem necessary to me. Not many people will need to look at this 
and the list of commands already is rather long.

 *) Permissions: Transforms don't have owners, a bit like casts.
 Currently, you are allowed to drop a transform if you own both the type
 and the language.  That might be too strict, maybe own the type and have
 privileges on the language would be enough.
Seems sensible enough to me.

 *) If we want to offer the ability to write transforms to end users,
 then we need to install all the header files for the languages and the
 types.  This actually worked quite well; including hstore.h and plperl.h
 for example, gave you what you needed.  In other cases, some headers
 might need cleaning up.  Also, some magic from the plperl and plpython
 build systems might need to be exposed, for example to find the header
 files.  See existing modules for how this currently looks.
Doesn't look to bad to me. I dont't know how this could be made easier.

 *) There is currently some syntax schizophrenia.  The grammar accepts
 
 CREATE TRANSFORM FOR hstore LANGUAGE plperl (
 FROM SQL WITH hstore_to_plperl,
 TO SQL WITH plperl_to_hstore
 );
 
 but pg_dump produces
 
 CREATE TRANSFORM FOR hstore LANGUAGE plperl (
 FROM SQL WITH hstore_to_plperl(hstore),
 TO SQL WITH plperl_to_hstore(internal)
 );
 
 The SQL standard allows both.  (In the same way that it allows 'DROP
 FUNCTION foo' without arguments, if it is not ambigious.)  Precedent is
 that CREATE CAST requires arguments, but CREATE LANGUAGE does not.
I don't find that problematic personally.

Greetings,

Andres
-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2012-06-16 Thread Jeff Janes
On Thu, Jun 14, 2012 at 3:42 PM, Peter Eisentraut pete...@gmx.net wrote:
 Here is my first patch for the transforms feature.  This is a mechanism
 to adapt data types to procedural languages.  The previous proposal was
 here: http://archives.postgresql.org/pgsql-hackers/2012-05/msg00728.php

When I apply this and go to contrib and do make check, I get:

In file included from hstore_plperl.c:4:0:
../../src/pl/plperl/plperl.h:49:20: fatal error: EXTERN.h: No such
file or directory

Cheers,

Jeff

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] transforms

2012-06-16 Thread Jeff Janes
On Sat, Jun 16, 2012 at 7:15 PM, Jeff Janes jeff.ja...@gmail.com wrote:
 On Thu, Jun 14, 2012 at 3:42 PM, Peter Eisentraut pete...@gmx.net wrote:
 Here is my first patch for the transforms feature.  This is a mechanism
 to adapt data types to procedural languages.  The previous proposal was
 here: http://archives.postgresql.org/pgsql-hackers/2012-05/msg00728.php

 When I apply this and go to contrib and do make check, I get:

 In file included from hstore_plperl.c:4:0:
 ../../src/pl/plperl/plperl.h:49:20: fatal error: EXTERN.h: No such
 file or directory

Ah, that went away when I remembered to ./configure --with-perl

Although the error message seem less than optimal.  Aren't test
usually skipped when they are missing prerequisites in the config?

Cheers,

Jeff

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers