Re: [HACKERS] Python 3.1 support
I wrote: On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. What is going to happen to plpython functions that depend on 2.x behavior? I have a proposal for how to handle this, and a prototype patch attached. This follows essentially what the CPython distribution itself does, which will make this tolerably easy to follow for users. We install plpython as plpython2.so or plpython3.so, depending on the version used to build it. Then, plpython.so is a symlink to plpython2.so. So here is the potentially final patch for this, including the original port of plpython.c itself, build system adjustments, and documentation. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Mon, Dec 14, 2009 at 1:42 PM, Peter Eisentraut pete...@gmx.net wrote: I wrote: On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. What is going to happen to plpython functions that depend on 2.x behavior? I have a proposal for how to handle this, and a prototype patch attached. This follows essentially what the CPython distribution itself does, which will make this tolerably easy to follow for users. We install plpython as plpython2.so or plpython3.so, depending on the version used to build it. Then, plpython.so is a symlink to plpython2.so. So here is the potentially final patch for this, including the original port of plpython.c itself, build system adjustments, and documentation. I think you forgot to actually attach it... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
I wrote:
On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
There was considerable debate earlier about whether we wanted to treat
Python 3 as a separate PL so it could be available in parallel with
plpython 2, because of the user-level coding incompatibilities. It
looks like this patch simply ignores that problem. What is going to
happen to plpython functions that depend on 2.x behavior?
I have a proposal for how to handle this, and a prototype patch
attached. This follows essentially what the CPython distribution itself
does, which will make this tolerably easy to follow for users.
We install plpython as plpython2.so or plpython3.so, depending on the
version used to build it. Then, plpython.so is a symlink to
plpython2.so.
So here is the potentially final patch for this, including the original
port of plpython.c itself, build system adjustments, and documentation.
Really attached this time.
diff --git a/config/python.m4 b/config/python.m4
index 9160a2b..32fff43 100644
--- a/config/python.m4
+++ b/config/python.m4
@@ -30,10 +30,12 @@ else
AC_MSG_ERROR([distutils module not found])
fi
AC_MSG_CHECKING([Python configuration directory])
+python_majorversion=`${PYTHON} -c import sys; print(sys.version[[0]])`
python_version=`${PYTHON} -c import sys; print(sys.version[[:3]])`
python_configdir=`${PYTHON} -c from distutils.sysconfig import get_python_lib as f; import os; print(os.path.join(f(plat_specific=1,standard_lib=1),'config'))`
python_includespec=`${PYTHON} -c import distutils.sysconfig; print('-I'+distutils.sysconfig.get_python_inc())`
+AC_SUBST(python_majorversion)[]dnl
AC_SUBST(python_version)[]dnl
AC_SUBST(python_configdir)[]dnl
AC_SUBST(python_includespec)[]dnl
diff --git a/configure b/configure
index 009a177..be51281 100755
--- a/configure
+++ b/configure
@@ -677,6 +677,7 @@ python_libdir
python_includespec
python_configdir
python_version
+python_majorversion
PYTHON
perl_embed_ldflags
perl_useshrplib
@@ -6964,6 +6965,7 @@ $as_echo $as_me: error: distutils module not found 2;}
fi
{ $as_echo $as_me:$LINENO: checking Python configuration directory 5
$as_echo_n checking Python configuration directory... 6; }
+python_majorversion=`${PYTHON} -c import sys; print(sys.version[0])`
python_version=`${PYTHON} -c import sys; print(sys.version[:3])`
python_configdir=`${PYTHON} -c from distutils.sysconfig import get_python_lib as f; import os; print(os.path.join(f(plat_specific=1,standard_lib=1),'config'))`
python_includespec=`${PYTHON} -c import distutils.sysconfig; print('-I'+distutils.sysconfig.get_python_inc())`
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 4746e68..f9221c9 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -195,8 +195,12 @@ su - postgres
para
To build the applicationPL/Python/ server programming
language, you need a productnamePython/productname
- installation with the header files and the applicationdistutils/application module.
- The minimum required version is productnamePython/productname 2.2.
+ installation with the header files and
+ the applicationdistutils/application module. The minimum
+ required version is productnamePython/productname
+ 2.2. productnamePython 3/productname is supported with
+ version 3.1 or later; but see xref linkend=plpython-python23
+ when using Python 3.
/para
para
diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index 502a5bc..e6c998a 100644
--- a/doc/src/sgml/plpython.sgml
+++ b/doc/src/sgml/plpython.sgml
@@ -14,7 +14,8 @@
para
To install PL/Python in a particular database, use
- literalcreatelang plpythonu replaceabledbname//literal.
+ literalcreatelang plpythonu replaceabledbname//literal (but
+ see also xref linkend=plpython-python23).
/para
tip
@@ -42,6 +43,112 @@
/para
/note
+ sect1 id=plpython-python23
+ titlePython 2 vs. Python 3/title
+
+ para
+ PL/Python supports both the Python 2 and Python 3 language
+ variants. (The PostgreSQL installation instructions might contain
+ more precise information about the exact supported minor versions
+ of Python.) Because the Python 2 and Python 3 language variants
+ are incompatible in some important aspects, the following naming
+ and transitioning scheme is used by PL/Python to avoid mixing them:
+
+ itemizedlist
+listitem
+ para
+ The PostgreSQL language named literalplpython2u/literal
+ implements PL/Python based on the Python 2 language variant.
+ /para
+/listitem
+
+listitem
+ para
+ The PostgreSQL language named literalplpython3u/literal
+ implements PL/Python based on the Python 3 language variant.
+ /para
+/listitem
+
+listitem
+ para
+ The language named literalplpythonu/literal implements
+ PL/Python based on the default Python language variant, which is
+ currently
Re: [HACKERS] Python 3.1 support
On Fri, 2009-12-11 at 01:19 +0200, Peter Eisentraut wrote: On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. What is going to happen to plpython functions that depend on 2.x behavior? I have a proposal for how to handle this, and a prototype patch attached. This follows essentially what the CPython distribution itself does, which will make this tolerably easy to follow for users. We install plpython as plpython2.so or plpython3.so, depending on the version used to build it. Then, plpython.so is a symlink to plpython2.so. We then create three language definition templates: plpythonu - plpython.so plpython2u - plpython2.so plpython3u - plpython3.so Comments? Well as a Python guy... makes sense to me :) Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
There was considerable debate earlier about whether we wanted to treat
Python 3 as a separate PL so it could be available in parallel with
plpython 2, because of the user-level coding incompatibilities. It
looks like this patch simply ignores that problem. What is going to
happen to plpython functions that depend on 2.x behavior?
I have a proposal for how to handle this, and a prototype patch
attached. This follows essentially what the CPython distribution itself
does, which will make this tolerably easy to follow for users.
We install plpython as plpython2.so or plpython3.so, depending on the
version used to build it. Then, plpython.so is a symlink to
plpython2.so.
We then create three language definition templates:
plpythonu - plpython.so
plpython2u - plpython2.so
plpython3u - plpython3.so
In the far future we flip the default symlink to plpython3.so, maybe in
about 5 years when Python 2.x expires.
This gives the users the following options and scenarios:
- Existing users don't have to do anything, until maybe in five years
they will notice that their OS doesn't ship Python 2 anymore and they
will have to act anyway. In practice, by then they might have adjusted
their coding style to Python 2.6/2.7 norms and their functions will
migrate to 3.x without change.
- Users who know that they have heavily Python 2.x dependent code and
don't want to ever change it can make use of the plpython2u language
name, just like they should probably change their scripts to use
something like #!/usr/bin/python2.
- Users who want to dive into Python 3 can use the plpython3u language
name, which will basically keep working forever by today's standards.
Those users would probably also use #!/usr/bin/python3 or the like in
their scripts. In the far future they might like to remove the 3.
- Daredevils can change symlink manually and make plpython3.so the
default plpythonu implementation. Those people would probably also
make /usr/bin/python be version 3.
Comments?
diff --git a/config/python.m4 b/config/python.m4
index 9160a2b..32fff43 100644
--- a/config/python.m4
+++ b/config/python.m4
@@ -30,10 +30,12 @@ else
AC_MSG_ERROR([distutils module not found])
fi
AC_MSG_CHECKING([Python configuration directory])
+python_majorversion=`${PYTHON} -c import sys; print(sys.version[[0]])`
python_version=`${PYTHON} -c import sys; print(sys.version[[:3]])`
python_configdir=`${PYTHON} -c from distutils.sysconfig import get_python_lib as f; import os; print(os.path.join(f(plat_specific=1,standard_lib=1),'config'))`
python_includespec=`${PYTHON} -c import distutils.sysconfig; print('-I'+distutils.sysconfig.get_python_inc())`
+AC_SUBST(python_majorversion)[]dnl
AC_SUBST(python_version)[]dnl
AC_SUBST(python_configdir)[]dnl
AC_SUBST(python_includespec)[]dnl
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index ca7f499..7a6e3a9 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -171,6 +171,7 @@ python_libdir = @python_libdir@
python_libspec = @python_libspec@
python_additional_libs = @python_additional_libs@
python_configdir = @python_configdir@
+python_majorversion = @python_majorversion@
python_version = @python_version@
krb_srvtab = @krb_srvtab@
diff --git a/src/include/catalog/pg_pltemplate.h b/src/include/catalog/pg_pltemplate.h
index 8cdedb4..cbb0a33 100644
--- a/src/include/catalog/pg_pltemplate.h
+++ b/src/include/catalog/pg_pltemplate.h
@@ -73,5 +73,7 @@ DATA(insert ( pltclu f f pltclu_call_handler _null_ _null_ $libdir/pltcl
DATA(insert ( plperl t t plperl_call_handler plperl_inline_handler plperl_validator $libdir/plperl _null_ ));
DATA(insert ( plperlu f f plperl_call_handler plperl_inline_handler plperl_validator $libdir/plperl _null_ ));
DATA(insert ( plpythonu f f plpython_call_handler _null_ _null_ $libdir/plpython _null_ ));
+DATA(insert ( plpython2u f f plpython_call_handler _null_ _null_ $libdir/plpython2 _null_ ));
+DATA(insert ( plpython3u f f plpython_call_handler _null_ _null_ $libdir/plpython3 _null_ ));
#endif /* PG_PLTEMPLATE_H */
diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index 840b874..d11525d 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -36,7 +36,7 @@ override CPPFLAGS := -I$(srcdir) $(python_includespec) $(CPPFLAGS)
rpathdir = $(python_libdir)
-NAME = plpython
+NAME = plpython$(python_majorversion)
OBJS = plpython.o
@@ -56,7 +56,10 @@ endif
SHLIB_LINK = $(python_libspec) $(python_additional_libs) $(filter -lintl,$(LIBS))
-REGRESS_OPTS = --dbname=$(PL_TESTDB) --load-language=plpythonu
+REGRESS_OPTS = --dbname=$(PL_TESTDB)
+ifeq ($(python_majorversion),2)
+REGRESS_OPTS += --load-language=plpythonu
+endif
REGRESS = \
plpython_schema \
plpython_populate \
@@ -83,10 +86,16 @@ include $(top_srcdir)/src/Makefile.shlib
all: all-lib
install: all installdirs install-lib
+ifeq ($(python_majorversion),2)
+ cd
Re: [HACKERS] Python 3.1 support
On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote: Is there any precedent for the sort of behavior that you are implementing, that is, automatic sharing of variables between independent executions of the same source container? import foo # bar is a regular, def'd function. foo.bar() ... # even in another thread, doesn't matter.. foo.bar() In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary). A plpython3 function *is* a Python module. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On fre, 2009-11-20 at 01:20 -0700, James Pye wrote: On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote: Is there any precedent for the sort of behavior that you are implementing, that is, automatic sharing of variables between independent executions of the same source container? import foo # bar is a regular, def'd function. foo.bar() ... # even in another thread, doesn't matter.. foo.bar() In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary). That's not what I meant, because this is the same execution of the same source container, with threads explicitly started somewhere. You could do the same in a plpython function (in theory, at least). What I mean is more like, you execute the same source file twice in a row, and the global variables are saved for the second run. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 20, 2009, at 1:26 AM, Peter Eisentraut wrote: because this is the same execution Hrm, not necessarily. foo could be imported by another, completely independent part of the program. foo is cached in sys.modules. bar() is executed and it's still the same globals(). shared. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Am 19.11.2009 18:01, schrieb James Pye: On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote: The other approach, which is what James Pye's new implementation proposes (as I understand it), is to convert PostgreSQL types into specially made Python objects, such as Postgres.types.record or Postgres.types.timestamp. Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum). So more like copied into Python memory, and associated with its respective type. Wrapped in a Python object? Yes wrapped is the term commonly used for that. And I must say I like it and I used plpy where I could. One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's no type I/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it could be quite beneficial. Exactly and you have all the meta information about the original postgres type. IIRC there were some thoughts of having something like that in a DBAPI interface as well (similar for example to cx_Oracle). Tino smime.p7s Description: S/MIME Cryptographic Signature
Re: [HACKERS] Python 3.1 support
On ons, 2009-11-18 at 09:48 -0800, Joshua D. Drake wrote: Although I wonder if longer term (2.x is going to be support a long time) we will end up with frustration within the single source file trying to keep things straight. There are five million Python modules with C code out there with the same problem. Considerable effort has been put in by Python upstream to make the effort manageable. No one in their right mind is going to create two separate source files just because in the future the mythical differences will be too big, when clearly the effort is going into a direction to reduce the differences. If you look into the source file, there is already special code for Python 2.2, 2.3, 2.4, 2.5, 2.6, and now 3.1. The chunk for 3.1 is a bit bigger, but only a bit, and well, that's why it's 3.x and not 2.x. No one has ever suggested, we might need to support Python 2.2 for a long time, let's create a separate source file. I agree, there will probably need to be some configuration/build support on top of this, but that's something we should work out independently of how to manage the source file. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On ons, 2009-11-18 at 08:43 -0800, Nathan Boley wrote: Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing. Well, FWIW, I am excited about better type integration. Let's clarify, as there are two different models being proposed here. The first approach, which is currently implemented (and some patches pending), is to convert a PostgreSQL type to the nearest Python type. For example, text to string, int to int, array to list, timestamp to datetime.datetime, etc. The other approach, which is what James Pye's new implementation proposes (as I understand it), is to convert PostgreSQL types into specially made Python objects, such as Postgres.types.record or Postgres.types.timestamp. Also, I am a little skeptical about this patch. I am sorry if this has already been discussed, but would this mean that I need to choose whether pl/python is built against Python 2.* or Python 3.*? Yeah, see later discussion about how to resolve this. But I think in practice, unless you use lots of print statements in your stored procedures (?!?), this problem is exaggerated. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On ons, 2009-11-18 at 11:32 -0800, Nathan Boley wrote: I took a cursory look at this patch and, while the logic seems sound and roughly in line with the suggested python porting procedure, I'm not quite certain what this implies for potential future patches. For instance, if I wanted to write a type converter for bytea - the python 3 byte type would the expectation be that I ensure that it works in Python 2? Or is an ifdef that ignores it in the case of Python 2 OK, and we can just put a note in the docs. Note that this is already implemented. The main point of the patch is to provide a small compatibility layer so that these kinds of issues are practically nonexistent. The fact that you didn't notice might prove that the patch does its job. ;-) Also, how far back do we want to maintain 2.x compatibility? 2.0? We handle this on an ad hoc basis. We currently support Python 2.2 and later, and this cutoff exists -- this is my interpretation of history -- because 2.2 introduced iterators and no one bothered(?) to put ifdefs around the code in PL/Python that provides iterator support. Over the years, we will probably drop support for other older Python versions, but there is no process or plan for that. Right now, the support for Python 2.2 is about three lines, so it's not a bother, but when someone comes and implements a major feature that, say, requires Python 2.3, we can probably drop 2.2. But when the major feature requires 2.6, we probably don't want to drop 2.5 quite yet at this time. It's a judgment call. If I wanted to submit a patch that makes use of the list sort method, do I need to ensure that it can either use the cmp arguments or a key argument? Any patch one is likely to submit will be a C patch, not a Python patch. But anyway, the key argument was introduced in Python 2.4, and so we'd have to come to a decision in the community about whether Python 2.3 support is worth keeping versus the value of that new feature. See above. But anyway, this problem has nothing to do with my patch; it has already existed in the same form forever. What if I wanted to implement a set returning function that made use of an iterators next() method. Would I just put ifdefs around the code or a preprocessor definition that defines NEXT as next() for Python 2.x and __next__() for 3.x? Again, you would likely submit a C patch, and the iterator API is the same between 2.x and 3.x. I guess that my first impression is that Python broke compatibility for a reason, and that either plpython can't evolve, or it will quickly become impossible to maintain. I think this is an exaggeration of reality. Python 3 removed deprecated features. There is a perfectly good migration path that covers most code: Switch to Python 2.6, switch to the new features, remove the old features, switch to Python 3.x. This applies both on the Python and the C level. They did not break compatibility with the intention of making every module author out there reimplement their thing from scratch. Otherwise Python 2.6 would make very little sense at all. Take a look at an example closer to home: PostgreSQL breaks C API compatibility in almost every major release. We do this to remove cruft and support new features. The intent is not to make Slony and PostGIS and all the other modules reimplement their product from scratch every time. They put in a few ifdefs, sometimes they complain about it ;-), and then the problem is solved. That being said, I mostly buy the maintenance arguments from the previous discussion, but if we want to have plpython and plpython3, a bunch of defines and ifdefs does not seem like the best way to do this. These ifdefs were not my idea. They are in some cases directly and in some cases in spirit from the Python 2.6 header files, so they are the official way to do this. Would a better approach be to maintain compatibility layer? ie plython_compat.h/c plython2.c plython3.c Then patches that apply to a python3 can be applied to plython3.c and any changed function can be ripped out of plython_compat and moved into plpython2. As I tried to explain above, we have always had a rolling feature model of sorts, even across various Python 2.x versions. If you want to try it out, you could take the current source and split it up into plpython22.c, plpython23.c, etc. and see if that becomes useful. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote: The other approach, which is what James Pye's new implementation proposes (as I understand it), is to convert PostgreSQL types into specially made Python objects, such as Postgres.types.record or Postgres.types.timestamp. Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum). So more like copied into Python memory, and associated with its respective type. Wrapped in a Python object? One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's no type I/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it could be quite beneficial. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On ons, 2009-11-18 at 12:28 -0500, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: Yes. That's exactly what I was complaining about upthread. I'm not a Python user, but from what I can gather of the 2-to-3 changes, having to choose one at package build time is going to be a disaster. Agreed. We really need to have a plpython and plpython3. Peter was concerned about duplicative maintenance effort, but what I think this patch shows is that (at least for the near future) both could be built from a single source file. What we need is configure and makefile support to do that. By the way, it occurred to me that having two different versions of libpython loaded into the same process is probably not going to work sanely. So whatever solution we come up with for the Python 3 transition, the possibilities for a jolly back-and-forth are probably going to be quite limited. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On ons, 2009-11-18 at 13:36 -0700, James Pye wrote: On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote: The question is whether it helps the user, not the implementer. Sure, but do you have a patch waiting to implement tracebacks? I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without some significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack. What helped the implementer here would help the user. But you wouldn't, for example, get away with breaking SQL (or even improving it incompatibly) to facilitate a better elog. As far as I can tell, it just creates more typing for no benefit whatsoever. def main(*args): is annoying, but not entirely lamentable... It's explicit, as well(no need to document munging that occurs behind the scenes). Also, compare the cases where you need to cache some initialized data: if 'key' not in SD: ... SD['key'] = my_newly_initialized_data ... With function modules, SD is not needed as you have your module globals to keep your locally cached data in: ... data = my_newly_initialized_data def main(*args): ... I can see that this creates other options for structuring code, but it doesn't actually match my way of thinking. (Obviously, I'm biased, but anyway.) I think of a PL/Python function as a Python script file stored in the database. When you call it, arguments are passed just like a Python script receives arguments from the shell. When Python scripts want to share data, they might use a file (or perhaps a database server in advanced cases) and do if not file exists: create the file fill it with data This is in my mind quite analogous to how the SD business works. The analogy to your approach, as I understand it, would be that multiple instances of the same script file will automatically share their global variables. That could be quite interesting, actually, but it's not how it works, and in most cases it's better that way. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Peter Eisentraut pete...@gmx.net writes: By the way, it occurred to me that having two different versions of libpython loaded into the same process is probably not going to work sanely. Why not? There's no way they'd even know about each other. We tell the loader not to make the symbols globally visible. But in any case, my main concern here is that I don't want to have to predetermine which python version a user of Red Hat/Fedora will have to use. If they can only use one at a time, that's still a good bit better than not having a choice at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 19, 2009, at 11:32 AM, Peter Eisentraut wrote: But you wouldn't, for example, get away with breaking SQL (or even improving it incompatibly) to facilitate a better elog. This doesn't fit the situation. I'm not breaking PL/Python. I'm trying to add PL/Python3. =) I think of a PL/Python function as a Python script file stored in the database. For Python, I think that's a mistake. Python scripts are independent applications. [tho, I think this does illuminate our perspectives...] -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On tor, 2009-11-19 at 13:43 -0500, Tom Lane wrote: But in any case, my main concern here is that I don't want to have to predetermine which python version a user of Red Hat/Fedora will have to use. If they can only use one at a time, that's still a good bit better than not having a choice at all. By the way, mod_wsgi supports Python 3 already (same patch as here, in principle). From the Fedora wiki page, I gather that no one has really looked into packaging that yet for Python 3, but if someone does, maybe we can cross-inspire ourselves. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On tor, 2009-11-19 at 13:12 -0700, James Pye wrote: I think of a PL/Python function as a Python script file stored in the database. For Python, I think that's a mistake. Python scripts are independent applications. Is there any precedent for the sort of behavior that you are implementing, that is, automatic sharing of variables between independent executions of the same source container? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On sön, 2009-11-15 at 18:39 -0700, James Pye wrote: I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around... The question is whether it helps the user, not the implementer. As far as I can tell, it just creates more typing for no benefit whatsoever. Also, it's inconsistent with normal Python script files and with other PLs. AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object. OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.) Right, if I use PL/Python, I do it because I want to use Python. I don't need another PostgreSQL implementation on top of Python. The maintenance effort required to keep those two consistent aside. Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing. the pain of dealing with a second implementation. What pain are you anticipating? Maintenance? Right. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing. Well, FWIW, I am excited about better type integration. Also, I am a little skeptical about this patch. I am sorry if this has already been discussed, but would this mean that I need to choose whether pl/python is built against Python 2.* or Python 3.*? -Nathan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Nathan Boley npbo...@gmail.com writes: Also, I am a little skeptical about this patch. I am sorry if this has already been discussed, but would this mean that I need to choose whether pl/python is built against Python 2.* or Python 3.*? Yes. That's exactly what I was complaining about upthread. I'm not a Python user, but from what I can gather of the 2-to-3 changes, having to choose one at package build time is going to be a disaster. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: Nathan Boley npbo...@gmail.com writes: Also, I am a little skeptical about this patch. I am sorry if this has already been discussed, but would this mean that I need to choose whether pl/python is built against Python 2.* or Python 3.*? Yes. That's exactly what I was complaining about upthread. I'm not a Python user, but from what I can gather of the 2-to-3 changes, having to choose one at package build time is going to be a disaster. Agreed. We really need to have a plpython and plpython3. Heck this would play nicely too because we support backward compatibility but also upward version differences. Joshua D. Drake regards, tom lane -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Joshua D. Drake j...@commandprompt.com writes: On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: Yes. That's exactly what I was complaining about upthread. I'm not a Python user, but from what I can gather of the 2-to-3 changes, having to choose one at package build time is going to be a disaster. Agreed. We really need to have a plpython and plpython3. Peter was concerned about duplicative maintenance effort, but what I think this patch shows is that (at least for the near future) both could be built from a single source file. What we need is configure and makefile support to do that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: Yes. That's exactly what I was complaining about upthread. I'm not a Python user, but from what I can gather of the 2-to-3 changes, having to choose one at package build time is going to be a disaster. Agreed. We really need to have a plpython and plpython3. Peter was concerned about duplicative maintenance effort, but what I think this patch shows is that (at least for the near future) both could be built from a single source file. What we need is configure and makefile support to do that. Ahh, so we would have: --enable-plpython2=/usr/bin/python2 --enable-plpython3=/usr/bin/python3 ? That seems reasonable if we can run both. Although I wonder if longer term (2.x is going to be support a long time) we will end up with frustration within the single source file trying to keep things straight. Joshua D. Drake regards, tom lane -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Joshua D. Drake j...@commandprompt.com writes: On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote: Peter was concerned about duplicative maintenance effort, but what I think this patch shows is that (at least for the near future) both could be built from a single source file. That seems reasonable if we can run both. Although I wonder if longer term (2.x is going to be support a long time) we will end up with frustration within the single source file trying to keep things straight. Once it gets to the point where it's more trouble to keep them together than not, we can split the source. But judging from this patch, a single source file is the ticket for the moment. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Here's the patch to support Python =3.1 with PL/Python. The compatibility code is mostly in line with the usual 2-3 C porting practice and is documented inline. I took a cursory look at this patch and, while the logic seems sound and roughly in line with the suggested python porting procedure, I'm not quite certain what this implies for potential future patches. For instance, if I wanted to write a type converter for bytea - the python 3 byte type would the expectation be that I ensure that it works in Python 2? Or is an ifdef that ignores it in the case of Python 2 OK, and we can just put a note in the docs. Also, how far back do we want to maintain 2.x compatibility? 2.0? If I wanted to submit a patch that makes use of the list sort method, do I need to ensure that it can either use the cmp arguments or a key argument? What if I wanted to implement a set returning function that made use of an iterators next() method. Would I just put ifdefs around the code or a preprocessor definition that defines NEXT as next() for Python 2.x and __next__() for 3.x? I guess that my first impression is that Python broke compatibility for a reason, and that either plpython can't evolve, or it will quickly become impossible to maintain. That being said, I mostly buy the maintenance arguments from the previous discussion, but if we want to have plpython and plpython3, a bunch of defines and ifdefs does not seem like the best way to do this. Would a better approach be to maintain compatibility layer? ie plython_compat.h/c plython2.c plython3.c Then patches that apply to a python3 can be applied to plython3.c and any changed function can be ripped out of plython_compat and moved into plpython2. I'm sorry to snipe from the sidelines like this. If we didn't expect plpython to evolve then this patch seems like the correct approach, but there is clearly some desire to expand plpython and following this path seems like it will end in a much more painful split in the future or a necessary rewrite. If there is some consensus that this is the best approach, then I will do a more comprehensive review. -Nathan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote: The question is whether it helps the user, not the implementer. Sure, but do you have a patch waiting to implement tracebacks? I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without some significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack. What helped the implementer here would help the user. As far as I can tell, it just creates more typing for no benefit whatsoever. def main(*args): is annoying, but not entirely lamentable... It's explicit, as well(no need to document munging that occurs behind the scenes). Also, compare the cases where you need to cache some initialized data: if 'key' not in SD: ... SD['key'] = my_newly_initialized_data ... With function modules, SD is not needed as you have your module globals to keep your locally cached data in: ... data = my_newly_initialized_data def main(*args): ... Also, it's inconsistent with normal Python script files Hinges on whether normal is actually normal. I often use the __name__ convention in script files myself: if __name__ == '__main__': main(...) That is, using that convention, the script can be import'd and used without executing the script functionality. (It has proven to be very handy a few times now) and with other PLs. I don't understand why that's a significant enough interest to note. I don't need another PostgreSQL implementation on top of Python. Indeed, and I do understand that. That is, I have removed some features with that very thought in mind. (OTOH, I consider the date_part properties on datetime types to be special: too likely useful.) [tho, PostgreSQL implementation? I think I understand what you were getting at, but..] The maintenance effort required to keep those two consistent aside. I don't think there are many consistency issues here. What did you have in mind? Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing. Agreed. It would seem quite doomed. At this point, I'm not going to try getting it into PG. (apparent futility and such) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 18, 2009, at 1:36 PM, James Pye wrote: At this point, I'm not going to try getting it into PG. (apparent futility and such) ugh, on second thought, I think I've written a bit too much code to stop now. I'm going to get plpython3 as far as I can and submit it to the next commitfest. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On fre, 2009-11-13 at 11:27 -0700, James Pye wrote: Some are TODOs, so in part by other people. Some were briefly touched on in the recent past discussions(around the time that I announced the WIP). Native typing vs conversion, function fragments vs function modules. I'm of course only one user, but these two features don't excite me at all, and certainly not enough to go through the pain of dealing with a second implementation. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 15, 2009, at 6:37 AM, Peter Eisentraut wrote: but these two features don't excite me at all, hrm.. at all? I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around... AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object. OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.) [looking at the PL/Python todo list..] Excepting DB-API and trusted, I believe all the current PL/Python TODOs are fulfilled or N/A in plpython3... ugh, the docs are not yet complete, but I like to think of them as better anyways. :P the pain of dealing with a second implementation. What pain are you anticipating? Maintenance? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: Here's the patch to support Python =3.1 with PL/Python. The compatibility code is mostly in line with the usual 2-3 C porting practice and is documented inline. There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. Exactly how to package that is something to be determined by the packagers, and we can give them the support they need. But first you need code that works with Python 3, which is what this patch does. What is going to happen to plpython functions that depend on 2.x behavior? The porting path from 2 to 3 is pretty well established. You first port to 2.6, then remove all the old features, then move to 3.x. This is not something we have to reinvent. The only significant differences that you can't use in 2.6 without future imports are unicode literals and the print function, both of which are not in common use in PL/Python. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On tor, 2009-11-12 at 18:42 -0700, James Pye wrote: For me, plpython has never been what I would call a pleasure to use, and many of the gripes that I have with it are, IMO, entrenched far enough into the implementation that any efforts to change it would(should? =) cause unacceptable breakage in user applications(?). Has this list of gripes ever been brought up and discussed in this forum? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 13, 2009, at 4:47 AM, Peter Eisentraut wrote: Has this list of gripes ever been brought up and discussed in this forum? Some are TODOs, so in part by other people. Some were briefly touched on in the recent past discussions(around the time that I announced the WIP). Native typing vs conversion, function fragments vs function modules. I don't think native typing has seen any actual discussion, but I do recall mentioning it as something that I wanted to do(implicitly griped?). ... There is a difference in the situation from the discussion before. Prior, it was, I would like to implement a new PL for Python 3 with these features, and now, it is, I have implemented a new PL for Python 3 with these features. Simply, -hackers can choose among moving forward with Python 3 support in plpython or going with plpython3 or even both, I suppose(?). Naturally, I'm biased toward something that involves plpython3, so I don't think I can(should?) be of much help to -hackers as a Python PG user in any such decision. Of course, excepting the provision of justifications for my implementation/design choices... I would really love to see some input from Python users. I certainly don't want to waste time trying to get something into pgsql that Python users don't want. [here's a gripe that I haven't brought up as I think it is a matter of taste] I find (plpython) trigger functions to be a bit odd. I think it's the way in which manipulation/suppression decisions are made in BEFORE ROW triggers(return OK, SKIP, etc).. [label this as opinion at this point as I have yet to be able to nail down what, specifically, is wrong or un-pythonic about them.] Also, having distinct entry points to handle trigger events helps reduce potential errors by forcing the user to explicitly state the events that the trigger function can handle. Currently, in plpython, users *should* do sanity checks. Function modules opened the door for implementing this in a natural way, multiple functions(entry points) in the function module. http://python.projects.postgresql.org/pldocs/plpython3-programming.html#PLPYTHON3-FUNCTIONS-TRIGGERS -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Peter Eisentraut pete...@gmx.net writes: Here's the patch to support Python =3.1 with PL/Python. The compatibility code is mostly in line with the usual 2-3 C porting practice and is documented inline. There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. What is going to happen to plpython functions that depend on 2.x behavior? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Nov 12, 2009, at 12:54 PM, Peter Eisentraut wrote: Here's the patch to support Python =3.1 with PL/Python. :\ I was hoping to be able to use Python 3 to draw a clear distinction between plpython and the would-be plpython3 that I've been working on. I understand that you're not in favor of a brand new implementation for Python 3. Despite my dislike for that position(well, it would seem to be in opposition to my initiative, so naturally =), I don't entirely disagree with your rationale[wrt doing things more incrementally]. For me, plpython has never been what I would call a pleasure to use, and many of the gripes that I have with it are, IMO, entrenched far enough into the implementation that any efforts to change it would(should? =) cause unacceptable breakage in user applications(?). Well, as far as additional Python interfaces are concerned, a lot of redundant functionality, but that's not even the half of it. [I was hoping to get to a status message this weekend, but it seems like I should follow-up here. =] So here's where I'm at: -- Mostly documentation improvements since I last pinged -hackers. Still, *sigh*, filling in documentation and fighting bugs as I go. Currently resolving a bug instantiating MD arrays from nested lists. Once I'm finished with the docs, I'm going to start looking for refcount leaks. No major additions or changes are planned, but I have been making some minor additions as I write more docs. Overview/Features: http://wiki.postgresql.org/wiki/WIP:plpython3 Documentation: http://python.projects.postgresql.org/pldocs/plpython3.html git repo[see the plpython3 branch]: http://git.postgresql.org/gitweb?p=plpython3.git;a=summary Most of the documented interfaces have tests. I only have two platforms at my disposal, so I do fear that this will not just work on all of PG's supported platforms. Specifically, I've ran the tests on freebsd/amd64 and Mac10.6/intel(of course 10.5 as well for some earlier revisions). [err, actually, it's been a while since I ran the tests on freebsd.] -- plpython3 is turning out to be kinda beefy(~974K diff[eh, there is some fluff in there]), and I can't say that I've seen much interest in it, so I can't really blame anyone if -hackers ends up taking a pass on it. (python3 is too far away for most folk to care? folk are content with plpython?) eh, cheers, either way. =) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
