Re: Resolving the python 2 -> python 3 mess
Noah Misch writes:
> On Wed, Feb 19, 2020 at 08:42:36PM +0100, Peter Eisentraut wrote:
>> I think there should just
>> be an option "plpython is: {2|3|don't build it at all}". Then packagers can
>> match this to what their plan for /usr/bin/python* is -- which appears to be
>> different everywhere.
> Today, we do not give packagers this sort of discretion over SQL-level
> behavior. We formerly had --disable-integer-datetimes, but I view that as the
> lesser of two evils, the greater of which would have been to force dump/reload
> of terabyte-scale clusters. We should continue to follow today's principle,
> which entails not inviting packagers to align the nature of "LANGUAGE
> plpythonu" with the nature of /usr/bin/python.
FWIW, I've abandoned this patch. We've concluded (by default, at least)
that nothing is getting done in v13, and by the time v14 is out it will
be too late to have any useful effect. I expect that the situation
on-the-ground by 2021 will be that packagers build with PYTHON=python3
and package whatever they get from that. That means (1) plpythonu won't
exist anymore and (2) users will be left to their own devices to convert
existing plpython code. Now, (1) corresponds to not providing any
/usr/bin/python executable, only python3 -- and that is a really common
choice for distros to make, AFAIK, so I don't feel too awful about it.
I find (2) less than ideal, but there's evidently not enough interest
in doing anything about it. There's certainly going to be no point in
shipping a solution for (2) if we fail to do so before v14; people
will already have done the work by hand.
We should, however, consider updating the plpython docs to reflect
current reality. Notably, the existing wording in section 45.1
suggests that we'll eventually redefine "plpythonu" as Python 3,
and it seems to me that that's not going to happen.
regards, tom lane
Re: Resolving the python 2 -> python 3 mess
On Wed, Feb 19, 2020 at 08:42:36PM +0100, Peter Eisentraut wrote:
> I think there should just
> be an option "plpython is: {2|3|don't build it at all}". Then packagers can
> match this to what their plan for /usr/bin/python* is -- which appears to be
> different everywhere.
Today, we do not give packagers this sort of discretion over SQL-level
behavior. We formerly had --disable-integer-datetimes, but I view that as the
lesser of two evils, the greater of which would have been to force dump/reload
of terabyte-scale clusters. We should continue to follow today's principle,
which entails not inviting packagers to align the nature of "LANGUAGE
plpythonu" with the nature of /usr/bin/python. Users wouldn't benefit from
alignment. Moreover, it has long been conventional for the host to be a VM
dedicated to PostgreSQL, in which case users of the DBMS aren't users of its
host. I don't have an opinion on which version "LANGUAGE plpythonu" should
denote in PostgreSQL v13, but the version should be constant across v13
configurations. (If some builds make it unavailable or make it an error-only
stub, that is no problem.)
I reviewed all code:
On Thu, Feb 27, 2020 at 04:11:05PM -0500, Tom Lane wrote:
> --- a/configure.in
> +++ b/configure.in
> @@ -766,6 +766,9 @@ PGAC_ARG_BOOL(with, python, no, [build Python modules
> (PL/Python)])
> AC_MSG_RESULT([$with_python])
> AC_SUBST(with_python)
>
> +PGAC_ARG_BOOL(with, python2-stub, no, [build Python 2 compatibility stub])
> +AC_SUBST(with_python2_stub)
> +
> #
> # GSSAPI
> #
> @@ -1042,6 +1045,12 @@ fi
> if test "$with_python" = yes; then
>PGAC_PATH_PYTHON
>PGAC_CHECK_PYTHON_EMBED_SETUP
> + # Disable building Python 2 stub if primary version isn't Python 3
> + if test "$python_majorversion" -lt 3; then
> +with_python2_stub=no
> + fi
Standard PostgreSQL practice would be to AC_MSG_ERROR in response to the
infeasible option, not to ignore the option.
> --- /dev/null
> +++ b/src/pl/plpython/sql/plpython_stub.sql
> +call convert_python3_all();
Building with PYTHON=python3 --with-python2-stub, this fails on my RHEL 7.8
installation. I had installed python34-tools, which provides /usr/bin/2to3-3.
I had not installed python-tools (v2.7.5), which provides /usr/bin/2to3.
Making a 2to3 -> 2to3-3 symlink let the test pass. Requiring such a step to
pass tests may or may not be fine; what do you think? (I am attaching
regression.diffs; to my knowledge, it's not interesting.)
> --- a/src/tools/msvc/config_default.pl
> +++ b/src/tools/msvc/config_default.pl
> @@ -16,6 +16,7 @@ our $config = {
> tcl => undef,# --with-tcl=
> perl => undef,# --with-perl=
> python=> undef,# --with-python=
> + python2_stub => 1, # --with-python2-stub (ignored unless Python is
> v3)
This default should not depend on whether one uses the MSVC build system or
uses the GNU make build system.
> --- /dev/null
> +++ b/src/pl/plpython/convert_python3--1.0.sql
> +create procedure convert_python3_all(tool text default '2to3',
> + options text default '')
> +language plpython3u as $$
> +import re, subprocess, tempfile
> +
> +# pattern to extract just the function header from pg_get_functiondef result
> +aspat = re.compile("^(.*?\nAS )", re.DOTALL)
This fails on:
create function convert1("
AS " int) returns int
AS $$return 123l$$
language
plpython2u
immutable;
That's not up to project standard, but I'm proceeding to ignore this since the
subject is an untrusted language and ~nobody uses such argument names.
diff -U3
/home/nm/src/pg/vp/3stub/src/pl/plpython/expected/python3/plpython_stub.out
/home/nm/src/pg/vp/3stub/src/pl/plpython/results/python3/plpython_stub.out
--- /home/nm/src/pg/vp/3stub/src/pl/plpython/expected/python3/plpython_stub.out
2020-05-28 01:01:35.445842589 -0700
+++ /home/nm/src/pg/vp/3stub/src/pl/plpython/results/python3/plpython_stub.out
2020-05-28 01:01:37.765861474 -0700
@@ -50,22 +50,33 @@
call convert_python3_all();
NOTICE: converting function convert1()
-NOTICE: converting function convert2()
+ERROR: subprocess.CalledProcessError: Command '2to3 --no-diffs -w
/tmp/tmpvmbr920w/temp.py' returned non-zero exit status 127.
+CONTEXT: Traceback (most recent call last):
+ PL/Python function "convert_python3_all", line 42, in
+subprocess.check_call(tool + " " + options + " --no-diffs -w " +
tmpdirname + "/temp.py", shell=True)
+ PL/Python function "convert_python3_all", line 310, in check_call
+PL/Python procedure "convert_python3_all"
\sf convert1()
CREATE OR REPLACE FUNCTION public.convert1()
RETURNS integer
- LANGUAGE plpython3u
+ LANGUAGE plpython2u
IMMUTABLE
-AS $function$return 123$function$
+AS $function$return 123l$function$
\sf convert2()
CREATE OR REPLACE FUNCTION public.convert2()
RETURNS integer
- LANGUAGE plpython3u
+ LANGUAGE plpythonu
IMMUTABLE
-AS $function$return 123$function$
+AS $function$return 123l$function$
-- clean
Re: Resolving the python 2 -> python 3 mess
I wrote: > [ a couple patches ] Ping? I wish somebody would review this. I'm not wedded to any of the details, but it would be an embarrassment for us to ship v13 without any response to the fact that Python 2 is EOL. regards, tom lane
Re: Resolving the python 2 -> python 3 mess
Marco Atzeri writes: > Am 17.02.2020 um 17:49 schrieb Tom Lane: >> 1. On platforms where Python 2.x is still supported, recommend that >> packagers continue to build both plpython2 and plpython3, same as now. > there is some documentation on how to build both ? > The INSTALL gives no hint. It's explained in the plpython documentation: basically you have to configure and build the source tree twice (although I think the second time you can just cd into src/pl/plpython and build/install only that much). regards, tom lane
Re: Resolving the python 2 -> python 3 mess
On 2020-03-26 06:46, Marco Atzeri wrote: Am 17.02.2020 um 17:49 schrieb Tom Lane: We've had multiple previous discussions of $SUBJECT (eg [1][2]), without any resolution of what to do exactly. Thinking about this some more, I had an idea that I don't think has been discussed. To wit: 1. On platforms where Python 2.x is still supported, recommend that packagers continue to build both plpython2 and plpython3, same as now. there is some documentation on how to build both ? You have to configure and build the sources twice with different PYTHON settings. It depends on your packaging system how to best arrange that. And how to build for multiples 3.x ? That is not supported. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Resolving the python 2 -> python 3 mess
Am 17.02.2020 um 17:49 schrieb Tom Lane: We've had multiple previous discussions of $SUBJECT (eg [1][2]), without any resolution of what to do exactly. Thinking about this some more, I had an idea that I don't think has been discussed. To wit: 1. On platforms where Python 2.x is still supported, recommend that packagers continue to build both plpython2 and plpython3, same as now. there is some documentation on how to build both ? The INSTALL gives no hint. And how to build for multiples 3.x ? Currently for Cygwin package I am building only 2.x and it is clearly not a good situation. Regards Marco
Re: Resolving the python 2 -> python 3 mess
Andrew Dunstan writes:
> On Thu, Feb 27, 2020 at 1:33 AM Tom Lane wrote:
>> OK, so we need that *and* the AddProject addition you mentioned?
> Yes, the first one builds it, the second one fixes the tests to run correctly.
Thanks, here's a patchset incorporating those fixes. Otherwise
same as before.
regards, tom lane
diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index 1921915..ac989a3 100644
--- a/doc/src/sgml/plpython.sgml
+++ b/doc/src/sgml/plpython.sgml
@@ -164,13 +164,6 @@
- See also the
- document https://docs.python.org/3/whatsnew/3.0.html;>What's
- New In Python 3.0 for more information about porting to
- Python 3.
-
-
-
It is not allowed to use PL/Python based on Python 2 and PL/Python
based on Python 3 in the same session, because the symbols in the
dynamic modules would clash, which could result in crashes of the
@@ -179,6 +172,90 @@
a mismatch is detected. It is possible, however, to use both
PL/Python variants in the same database, from separate sessions.
+
+
+ Converting from Python 2 to Python 3
+
+
+See the
+document https://docs.python.org/3/whatsnew/3.0.html;>What's
+New In Python 3.0 for the Python community's information and
+recommendations about porting to Python 3.
+
+
+
+PostgreSQL provides some support for helping
+you to convert existing Python 2 routines to Python 3. In an
+installation built with Python 3, there is an
+extension convert_python3 that changes functions
+and procedures from the plpythonu
+and plpython2u languages to
+the plpython3u language. While doing so, it applies
+the 2to3 tool described in the above document to
+the body of each such routine.
+
+
+
+Using convert_python3 can be as simple as:
+
+CREATE EXTENSION convert_python3;
+CALL convert_python3_all();
+
+This must be done as database superuser. If you wish, you can drop the
+extension once you're done converting everything.
+
+
+
+Since convert_python3 is Python 3 code, be careful
+not to install or run it in a session that has previously executed any
+Python 2 code. As explained above, that won't work.
+
+
+
+convert_python3_all has two optional arguments: the
+name of the conversion tool to use (by default 2to3,
+but you might for instance need to provide a full path name) and any
+special command-line options to provide to it. You might for example
+want to adjust the set of fixer rules
+that 2to3 applies:
+
+CALL convert_python3_all(options = '-f idioms -x apply');
+
+See 2to3's
+https://docs.python.org/3/library/2to3.html;>documentation
+for more information.
+
+
+
+The convert_python3 extension also provides a
+procedure that converts just one Python 2 function at a time:
+
+CALL convert_python3_one('myfunc(int)');
+
+The argument is the target function's OID, which can be written as
+a regprocedure constant (see
+). The main reason to use this would be
+if you need to use different options for different functions. It has
+the same optional arguments as convert_python3_all:
+
+CALL convert_python3_one('otherfunc(text)', tool = '/usr/bin/2to3',
+ options = '-f idioms');
+
+
+
+
+If you have needs that go beyond this, consult the source code for
+the convert_python3 extension (it's just a
+couple of plpython3u procedures) and adapt those
+procedures as necessary.
+
+
+
+Keep in mind that if you've constructed any DO blocks
+that use Python 2 code, those will have to be fixed up manually,
+wherever the source code for them exists.
+
+
diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index 9e95285..03f858b 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -38,6 +38,9 @@ DATA = $(NAME)u.control $(NAME)u--1.0.sql
ifeq ($(python_majorversion),2)
DATA += plpythonu.control plpythonu--1.0.sql
endif
+ifeq ($(python_majorversion),3)
+DATA += convert_python3.control convert_python3--1.0.sql
+endif
# header files to install - it's not clear which of these might be needed
# so install them all.
diff --git a/src/pl/plpython/convert_python3--1.0.sql b/src/pl/plpython/convert_python3--1.0.sql
new file mode 100644
index 000..3444ac0
--- /dev/null
+++ b/src/pl/plpython/convert_python3--1.0.sql
@@ -0,0 +1,149 @@
+/* convert_python3--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION convert_python3" to load this file. \quit
+
+-- This module provides two procedures, one to convert all python2
+-- functions and one to do just one. They're nearly identical, and
+-- in principle convert_python3_all() could be written as a loop
+-- around convert_python3_one(). It's not done that way since
+-- creating a temp directory for each function in a bulk
Re: Resolving the python 2 -> python 3 mess
On Thu, Feb 27, 2020 at 1:33 AM Tom Lane wrote: > > Andrew Dunstan writes: > > This seems to fix it. > > OK, so we need that *and* the AddProject addition you mentioned? > > Yes, the first one builds it, the second one fixes the tests to run correctly. cheers andrew -- Andrew Dunstanhttps://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Resolving the python 2 -> python 3 mess
Andrew Dunstan writes: > This seems to fix it. OK, so we need that *and* the AddProject addition you mentioned? regards, tom lane
Re: Resolving the python 2 -> python 3 mess
On 2/26/20 3:17 AM, Andrew Dunstan wrote:
> On 2/26/20 2:47 AM, Andrew Dunstan wrote:
>> On 2/25/20 8:24 PM, Andrew Dunstan wrote:
>>> On 2/25/20 7:08 PM, Tom Lane wrote:
Andrew Dunstan writes:
> On 2/25/20 5:06 PM, Tom Lane wrote:
>> No joy there --- now that I look closer, it seems the cfbot doesn't
>> build any of the external-language PLs on Windows. I'll have to
>> wait for some reviewer to try it.
> What are the requirements for testing? bowerbird builds with python 2.7,
> although I guess I should really try to upgrade it 3.x.
Has to be python 3, unfortunately; the patch has no effect on a
python 2 build.
>>> Yeah, I have python3 working on drongo, I'll test there.
>>>
>> It's almost there, you need to add something like this to Mkvcbuild.pm:
>>
>>
>> if ($solution->{options}->{python2_stub})
>> {
>> my $plpython2_stub =
>> $solution->AddProject('plpython2', 'dll', 'PLs',
>> 'src/pl/stub_plpython2');
>> $plpython2_stub->AddReference($postgres);
>> }
>
>
>
>
> However, when it get to testing contrib it complains like this:
>
>
>
> Checking hstore_plpython
> C:/prog/bf/root/HEAD/pgsql/Release/pg_regress/pg_regress
> --bindir=C:/prog/bf/root/HEAD/pgsql/Release/psql
> --dbname=contrib_regression --load-ex
> tension=hstore --load-extension=plpythonu
> --load-extension=hstore_plpythonu hstore_plpython
> (using postmaster on localhost, default port)
> == dropping database "contrib_regression" ==
> DROP DATABASE
> == creating database "contrib_regression" ==
> CREATE DATABASE
> ALTER DATABASE
> == installing hstore ==
> CREATE EXTENSION
> == installing plpythonu ==
> CREATE EXTENSION
> == installing hstore_plpythonu ==
> ERROR: could not access file "$libdir/hstore_plpython2": No such file
> or directory
> command failed: "C:/prog/bf/root/HEAD/pgsql/Release/psql/psql" -X -c
> "CREATE EXTENSION IF NOT EXISTS \"hstore_plpythonu\"" "contrib_regression"
>
>
> So there's a bit more work to do.
>
>
This seems to fix it.
cheers
andrew
--
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 82dca29a61..d3a07f84bb 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -423,15 +423,15 @@ sub subdircheck
@opts = grep { $_ !~ /plpythonu/ } @opts;
- if (-d "$topdir/$Config/plpython2")
+ if (-d "$topdir/$Config/plpython3")
{
- push @opts, "--load-extension=plpythonu";
- push @opts, '--load-extension=' . $module . 'u';
+ @tests = mangle_plpython3(\@tests);
}
- else
+ elsif (-d "$topdir/$Config/plpython2")
{
- # must be python 3
- @tests = mangle_plpython3(\@tests);
+ # if python3 doesn't exist this isn't the stub module
+ push @opts, "--load-extension=plpythonu";
+ push @opts, '--load-extension=' . $module . 'u';
}
}
Re: Resolving the python 2 -> python 3 mess
On 2/26/20 2:47 AM, Andrew Dunstan wrote:
> On 2/25/20 8:24 PM, Andrew Dunstan wrote:
>> On 2/25/20 7:08 PM, Tom Lane wrote:
>>> Andrew Dunstan writes:
On 2/25/20 5:06 PM, Tom Lane wrote:
> No joy there --- now that I look closer, it seems the cfbot doesn't
> build any of the external-language PLs on Windows. I'll have to
> wait for some reviewer to try it.
What are the requirements for testing? bowerbird builds with python 2.7,
although I guess I should really try to upgrade it 3.x.
>>> Has to be python 3, unfortunately; the patch has no effect on a
>>> python 2 build.
>>>
>>>
>>
>> Yeah, I have python3 working on drongo, I'll test there.
>>
> It's almost there, you need to add something like this to Mkvcbuild.pm:
>
>
> if ($solution->{options}->{python2_stub})
> {
> my $plpython2_stub =
> $solution->AddProject('plpython2', 'dll', 'PLs',
> 'src/pl/stub_plpython2');
> $plpython2_stub->AddReference($postgres);
> }
However, when it get to testing contrib it complains like this:
Checking hstore_plpython
C:/prog/bf/root/HEAD/pgsql/Release/pg_regress/pg_regress
--bindir=C:/prog/bf/root/HEAD/pgsql/Release/psql
--dbname=contrib_regression --load-ex
tension=hstore --load-extension=plpythonu
--load-extension=hstore_plpythonu hstore_plpython
(using postmaster on localhost, default port)
== dropping database "contrib_regression" ==
DROP DATABASE
== creating database "contrib_regression" ==
CREATE DATABASE
ALTER DATABASE
== installing hstore ==
CREATE EXTENSION
== installing plpythonu ==
CREATE EXTENSION
== installing hstore_plpythonu ==
ERROR: could not access file "$libdir/hstore_plpython2": No such file
or directory
command failed: "C:/prog/bf/root/HEAD/pgsql/Release/psql/psql" -X -c
"CREATE EXTENSION IF NOT EXISTS \"hstore_plpythonu\"" "contrib_regression"
So there's a bit more work to do.
cheers
andrew
--
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Resolving the python 2 -> python 3 mess
On 2/25/20 8:24 PM, Andrew Dunstan wrote:
> On 2/25/20 7:08 PM, Tom Lane wrote:
>> Andrew Dunstan writes:
>>> On 2/25/20 5:06 PM, Tom Lane wrote:
No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.
>>> What are the requirements for testing? bowerbird builds with python 2.7,
>>> although I guess I should really try to upgrade it 3.x.
>> Has to be python 3, unfortunately; the patch has no effect on a
>> python 2 build.
>>
>>
>
>
> Yeah, I have python3 working on drongo, I'll test there.
>
It's almost there, you need to add something like this to Mkvcbuild.pm:
if ($solution->{options}->{python2_stub})
{
my $plpython2_stub =
$solution->AddProject('plpython2', 'dll', 'PLs',
'src/pl/stub_plpython2');
$plpython2_stub->AddReference($postgres);
}
cheers
andrew
--
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Resolving the python 2 -> python 3 mess
On 2/25/20 7:08 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 2/25/20 5:06 PM, Tom Lane wrote: >>> No joy there --- now that I look closer, it seems the cfbot doesn't >>> build any of the external-language PLs on Windows. I'll have to >>> wait for some reviewer to try it. >> What are the requirements for testing? bowerbird builds with python 2.7, >> although I guess I should really try to upgrade it 3.x. > Has to be python 3, unfortunately; the patch has no effect on a > python 2 build. > > Yeah, I have python3 working on drongo, I'll test there. cheers andrew -- Andrew Dunstanhttps://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Resolving the python 2 -> python 3 mess
Andrew Dunstan writes: > On 2/25/20 5:06 PM, Tom Lane wrote: >> No joy there --- now that I look closer, it seems the cfbot doesn't >> build any of the external-language PLs on Windows. I'll have to >> wait for some reviewer to try it. > What are the requirements for testing? bowerbird builds with python 2.7, > although I guess I should really try to upgrade it 3.x. Has to be python 3, unfortunately; the patch has no effect on a python 2 build. regards, tom lane
Re: Resolving the python 2 -> python 3 mess
On 2/25/20 5:06 PM, Tom Lane wrote: > I wrote: >> I set up the MSVC scripts to default to building the stub extension. >> I don't know if we really want to commit it that way, but the idea >> for the moment is to try to get the cfbot to test it on Windows. > No joy there --- now that I look closer, it seems the cfbot doesn't > build any of the external-language PLs on Windows. I'll have to > wait for some reviewer to try it. > > What are the requirements for testing? bowerbird builds with python 2.7, although I guess I should really try to upgrade it 3.x. cheers andrew -- Andrew Dunstanhttps://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Resolving the python 2 -> python 3 mess
I wrote: > I set up the MSVC scripts to default to building the stub extension. > I don't know if we really want to commit it that way, but the idea > for the moment is to try to get the cfbot to test it on Windows. No joy there --- now that I look closer, it seems the cfbot doesn't build any of the external-language PLs on Windows. I'll have to wait for some reviewer to try it. regards, tom lane
Re: Resolving the python 2 -> python 3 mess
I wrote:
> Here's an updated pair of patches that attempt to fix the MSVC
> scripts (pretty blindly) and provide a very simple regression test.
A little *too* blindly, evidently. Try again ...
regards, tom lane
diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index 1921915..ac989a3 100644
--- a/doc/src/sgml/plpython.sgml
+++ b/doc/src/sgml/plpython.sgml
@@ -164,13 +164,6 @@
- See also the
- document https://docs.python.org/3/whatsnew/3.0.html;>What's
- New In Python 3.0 for more information about porting to
- Python 3.
-
-
-
It is not allowed to use PL/Python based on Python 2 and PL/Python
based on Python 3 in the same session, because the symbols in the
dynamic modules would clash, which could result in crashes of the
@@ -179,6 +172,90 @@
a mismatch is detected. It is possible, however, to use both
PL/Python variants in the same database, from separate sessions.
+
+
+ Converting from Python 2 to Python 3
+
+
+See the
+document https://docs.python.org/3/whatsnew/3.0.html;>What's
+New In Python 3.0 for the Python community's information and
+recommendations about porting to Python 3.
+
+
+
+PostgreSQL provides some support for helping
+you to convert existing Python 2 routines to Python 3. In an
+installation built with Python 3, there is an
+extension convert_python3 that changes functions
+and procedures from the plpythonu
+and plpython2u languages to
+the plpython3u language. While doing so, it applies
+the 2to3 tool described in the above document to
+the body of each such routine.
+
+
+
+Using convert_python3 can be as simple as:
+
+CREATE EXTENSION convert_python3;
+CALL convert_python3_all();
+
+This must be done as database superuser. If you wish, you can drop the
+extension once you're done converting everything.
+
+
+
+Since convert_python3 is Python 3 code, be careful
+not to install or run it in a session that has previously executed any
+Python 2 code. As explained above, that won't work.
+
+
+
+convert_python3_all has two optional arguments: the
+name of the conversion tool to use (by default 2to3,
+but you might for instance need to provide a full path name) and any
+special command-line options to provide to it. You might for example
+want to adjust the set of fixer rules
+that 2to3 applies:
+
+CALL convert_python3_all(options = '-f idioms -x apply');
+
+See 2to3's
+https://docs.python.org/3/library/2to3.html;>documentation
+for more information.
+
+
+
+The convert_python3 extension also provides a
+procedure that converts just one Python 2 function at a time:
+
+CALL convert_python3_one('myfunc(int)');
+
+The argument is the target function's OID, which can be written as
+a regprocedure constant (see
+). The main reason to use this would be
+if you need to use different options for different functions. It has
+the same optional arguments as convert_python3_all:
+
+CALL convert_python3_one('otherfunc(text)', tool = '/usr/bin/2to3',
+ options = '-f idioms');
+
+
+
+
+If you have needs that go beyond this, consult the source code for
+the convert_python3 extension (it's just a
+couple of plpython3u procedures) and adapt those
+procedures as necessary.
+
+
+
+Keep in mind that if you've constructed any DO blocks
+that use Python 2 code, those will have to be fixed up manually,
+wherever the source code for them exists.
+
+
diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index 9e95285..03f858b 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -38,6 +38,9 @@ DATA = $(NAME)u.control $(NAME)u--1.0.sql
ifeq ($(python_majorversion),2)
DATA += plpythonu.control plpythonu--1.0.sql
endif
+ifeq ($(python_majorversion),3)
+DATA += convert_python3.control convert_python3--1.0.sql
+endif
# header files to install - it's not clear which of these might be needed
# so install them all.
diff --git a/src/pl/plpython/convert_python3--1.0.sql b/src/pl/plpython/convert_python3--1.0.sql
new file mode 100644
index 000..3444ac0
--- /dev/null
+++ b/src/pl/plpython/convert_python3--1.0.sql
@@ -0,0 +1,149 @@
+/* convert_python3--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION convert_python3" to load this file. \quit
+
+-- This module provides two procedures, one to convert all python2
+-- functions and one to do just one. They're nearly identical, and
+-- in principle convert_python3_all() could be written as a loop
+-- around convert_python3_one(). It's not done that way since
+-- creating a temp directory for each function in a bulk conversion
+-- could get expensive.
+
+-- For some benighted reason, lib2to3 has exactly no documented API,
+-- so
Re: Resolving the python 2 -> python 3 mess
Here's an updated pair of patches that attempt to fix the MSVC
scripts (pretty blindly) and provide a very simple regression test.
I'm not too sure whether the regression test will really prove
workable or not: for starters, it'll fail if "2to3" isn't available
in the PATH. Perhaps there's reason to object to even trying to
test that, on security grounds.
I set up the MSVC scripts to default to building the stub extension.
I don't know if we really want to commit it that way, but the idea
for the moment is to try to get the cfbot to test it on Windows.
regards, tom lane
diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index 1921915..ac989a3 100644
--- a/doc/src/sgml/plpython.sgml
+++ b/doc/src/sgml/plpython.sgml
@@ -164,13 +164,6 @@
- See also the
- document https://docs.python.org/3/whatsnew/3.0.html;>What's
- New In Python 3.0 for more information about porting to
- Python 3.
-
-
-
It is not allowed to use PL/Python based on Python 2 and PL/Python
based on Python 3 in the same session, because the symbols in the
dynamic modules would clash, which could result in crashes of the
@@ -179,6 +172,90 @@
a mismatch is detected. It is possible, however, to use both
PL/Python variants in the same database, from separate sessions.
+
+
+ Converting from Python 2 to Python 3
+
+
+See the
+document https://docs.python.org/3/whatsnew/3.0.html;>What's
+New In Python 3.0 for the Python community's information and
+recommendations about porting to Python 3.
+
+
+
+PostgreSQL provides some support for helping
+you to convert existing Python 2 routines to Python 3. In an
+installation built with Python 3, there is an
+extension convert_python3 that changes functions
+and procedures from the plpythonu
+and plpython2u languages to
+the plpython3u language. While doing so, it applies
+the 2to3 tool described in the above document to
+the body of each such routine.
+
+
+
+Using convert_python3 can be as simple as:
+
+CREATE EXTENSION convert_python3;
+CALL convert_python3_all();
+
+This must be done as database superuser. If you wish, you can drop the
+extension once you're done converting everything.
+
+
+
+Since convert_python3 is Python 3 code, be careful
+not to install or run it in a session that has previously executed any
+Python 2 code. As explained above, that won't work.
+
+
+
+convert_python3_all has two optional arguments: the
+name of the conversion tool to use (by default 2to3,
+but you might for instance need to provide a full path name) and any
+special command-line options to provide to it. You might for example
+want to adjust the set of fixer rules
+that 2to3 applies:
+
+CALL convert_python3_all(options = '-f idioms -x apply');
+
+See 2to3's
+https://docs.python.org/3/library/2to3.html;>documentation
+for more information.
+
+
+
+The convert_python3 extension also provides a
+procedure that converts just one Python 2 function at a time:
+
+CALL convert_python3_one('myfunc(int)');
+
+The argument is the target function's OID, which can be written as
+a regprocedure constant (see
+). The main reason to use this would be
+if you need to use different options for different functions. It has
+the same optional arguments as convert_python3_all:
+
+CALL convert_python3_one('otherfunc(text)', tool = '/usr/bin/2to3',
+ options = '-f idioms');
+
+
+
+
+If you have needs that go beyond this, consult the source code for
+the convert_python3 extension (it's just a
+couple of plpython3u procedures) and adapt those
+procedures as necessary.
+
+
+
+Keep in mind that if you've constructed any DO blocks
+that use Python 2 code, those will have to be fixed up manually,
+wherever the source code for them exists.
+
+
diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index 9e95285..03f858b 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -38,6 +38,9 @@ DATA = $(NAME)u.control $(NAME)u--1.0.sql
ifeq ($(python_majorversion),2)
DATA += plpythonu.control plpythonu--1.0.sql
endif
+ifeq ($(python_majorversion),3)
+DATA += convert_python3.control convert_python3--1.0.sql
+endif
# header files to install - it's not clear which of these might be needed
# so install them all.
diff --git a/src/pl/plpython/convert_python3--1.0.sql b/src/pl/plpython/convert_python3--1.0.sql
new file mode 100644
index 000..3444ac0
--- /dev/null
+++ b/src/pl/plpython/convert_python3--1.0.sql
@@ -0,0 +1,149 @@
+/* convert_python3--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION convert_python3" to load this file. \quit
+
+-- This module provides two procedures, one to convert all python2
+--
Re: Resolving the python 2 -> python 3 mess
Peter Eisentraut writes: > Your scheme appears to center around the assumption that people will > want to port their functions at the same time as not building plpython2u > anymore. Not really; use of the proposed porting infrastructure is the same whether plpython2u still works or not. You end up with functions that are labeled plpython3u, so what bare "plpythonu" means is not a factor. It is true that as this patch is written, switching of plpythonu to point at Python 3 rather than 2 is coupled to disabling plpython2u. If we'd have gotten this done a year or two ago, I'd have made it more complex to allow more separation there. But events have passed us by: the info we are getting from packagers is that Python 2 is getting dropped *this year*, not in some distant future. So I think that allowing the plpythonu redefinition to be separate is no longer of any great value, and not worth extra complication for. People are just going to be shipping v13 with both things changed in any case. If we wanted to do something to help people port their functions in advance of the big changeover, the thing to do would be to back-patch the proposed convert_python3 extension into existing branches. regards, tom lane
Re: Resolving the python 2 -> python 3 mess
On 2020-02-19 05:39, Tom Lane wrote:
After thinking about this awhile longer, I'm starting to believe
we should do some of each. That is, the stub replacement for
plpython2.so should redirect "plpythonu" functions to plpython3.so,
but throw errors for "plpython2u" functions.
I'm not sure these complications are worth it. They don't match
anything that is done in other Python 2/3 porting schemes. I think
there should just be an option "plpython is: {2|3|don't build it at
all}". Then packagers can match this to what their plan for
/usr/bin/python* is -- which appears to be different everywhere.
Your scheme appears to center around the assumption that people will
want to port their functions at the same time as not building plpython2u
anymore. This would defeat testing functions before and after in the
same installation. I think the decisions of what plpythonu points to
and which variants are built at all should be separate.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Resolving the python 2 -> python 3 mess
After thinking about this awhile longer, I'm starting to believe
we should do some of each. That is, the stub replacement for
plpython2.so should redirect "plpythonu" functions to plpython3.so,
but throw errors for "plpython2u" functions. This is not because
of any technical difference between plpythonu and plpython2u ---
up to now, there wasn't any --- but because it seems like users
would be expecting that if they've read what we have said in
https://www.postgresql.org/docs/current/plpython-python23.html
Admittedly, what it says there is that plpythonu might become
Python 3 in some "distant" future release, not next year.
But at least there's a direct line between that documentation
and this behavior.
So attached is a pair of draft patches that do it like that.
0001 creates an extension with two conversion functions, based
on the script I showed in the other thread. Almost independently
of that, 0002 provides code to generate a stub version of
plpython2.so that behaves as stated above. 0002 is incomplete,
because I haven't looked into what is needed in the MSVC build
scripts. Maybe we could create some regression tests, too.
But I think these are potentially committable with those additions,
if people approve of this approach.
regards, tom lane
diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index 1921915..ac989a3 100644
--- a/doc/src/sgml/plpython.sgml
+++ b/doc/src/sgml/plpython.sgml
@@ -164,13 +164,6 @@
- See also the
- document https://docs.python.org/3/whatsnew/3.0.html;>What's
- New In Python 3.0 for more information about porting to
- Python 3.
-
-
-
It is not allowed to use PL/Python based on Python 2 and PL/Python
based on Python 3 in the same session, because the symbols in the
dynamic modules would clash, which could result in crashes of the
@@ -179,6 +172,90 @@
a mismatch is detected. It is possible, however, to use both
PL/Python variants in the same database, from separate sessions.
+
+
+ Converting from Python 2 to Python 3
+
+
+See the
+document https://docs.python.org/3/whatsnew/3.0.html;>What's
+New In Python 3.0 for the Python community's information and
+recommendations about porting to Python 3.
+
+
+
+PostgreSQL provides some support for helping
+you to convert existing Python 2 routines to Python 3. In an
+installation built with Python 3, there is an
+extension convert_python3 that changes functions
+and procedures from the plpythonu
+and plpython2u languages to
+the plpython3u language. While doing so, it applies
+the 2to3 tool described in the above document to
+the body of each such routine.
+
+
+
+Using convert_python3 can be as simple as:
+
+CREATE EXTENSION convert_python3;
+CALL convert_python3_all();
+
+This must be done as database superuser. If you wish, you can drop the
+extension once you're done converting everything.
+
+
+
+Since convert_python3 is Python 3 code, be careful
+not to install or run it in a session that has previously executed any
+Python 2 code. As explained above, that won't work.
+
+
+
+convert_python3_all has two optional arguments: the
+name of the conversion tool to use (by default 2to3,
+but you might for instance need to provide a full path name) and any
+special command-line options to provide to it. You might for example
+want to adjust the set of fixer rules
+that 2to3 applies:
+
+CALL convert_python3_all(options = '-f idioms -x apply');
+
+See 2to3's
+https://docs.python.org/3/library/2to3.html;>documentation
+for more information.
+
+
+
+The convert_python3 extension also provides a
+procedure that converts just one Python 2 function at a time:
+
+CALL convert_python3_one('myfunc(int)');
+
+The argument is the target function's OID, which can be written as
+a regprocedure constant (see
+). The main reason to use this would be
+if you need to use different options for different functions. It has
+the same optional arguments as convert_python3_all:
+
+CALL convert_python3_one('otherfunc(text)', tool = '/usr/bin/2to3',
+ options = '-f idioms');
+
+
+
+
+If you have needs that go beyond this, consult the source code for
+the convert_python3 extension (it's just a
+couple of plpython3u procedures) and adapt those
+procedures as necessary.
+
+
+
+Keep in mind that if you've constructed any DO blocks
+that use Python 2 code, those will have to be fixed up manually,
+wherever the source code for them exists.
+
+
diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index 0d53d3d..f0de8ff 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -38,6 +38,9 @@ DATA = $(NAME)u.control $(NAME)u--1.0.sql $(NAME)u--unpackaged--1.0.sql
ifeq ($(python_majorversion),2)
Re: Resolving the python 2 -> python 3 mess
> > So, as with Jesse's example, what I'm wondering is whether or not 2to3 > will fix that for you (or even flag it). The basic difference between > the two alternatives I suggested is whether we force people to put their > python function through that converter before we'll even try to run it. > Subtleties that 2to3 doesn't catch seem like non-reasons to insist on > applying it. > The 2018 vintage of 2to3 didn't catch it. It's not firsthand knowledge, but I just watched a nearby team have some production issues where one library couldn't fetch b'http://foo.org' so I'm guessing 2to3 still doesn't catch those things, or they stopped using it.
Re: Resolving the python 2 -> python 3 mess
Corey Huinker writes: >> A possible gotcha in this approach is if there are any python 2/3 >> incompatibilities that would not manifest as syntax errors or >> obvious runtime errors, but would allow old code to execute and >> silently do the wrong thing. > Unfortunately, I think there are cases like that. The shift to Unicode as > the default string means that some functions that used to return a `str` > now return a `bytes` (I know of this in the hashlib and base64 modules, but > probably also in URL request data and others), and to use a `bytes` in > string manipulation you have to first explicitly convert it to some string > encoding. So things like a function that wraps around a python crypto > library would be the exact places where those was-str-now-bytes functions > would be used. So, as with Jesse's example, what I'm wondering is whether or not 2to3 will fix that for you (or even flag it). The basic difference between the two alternatives I suggested is whether we force people to put their python function through that converter before we'll even try to run it. Subtleties that 2to3 doesn't catch seem like non-reasons to insist on applying it. regards, tom lane
Re: Resolving the python 2 -> python 3 mess
> > A possible gotcha in this approach is if there are any python 2/3 > incompatibilities that would not manifest as syntax errors or > obvious runtime errors, but would allow old code to execute and > silently do the wrong thing. One would hope that the Python crowd > weren't dumb enough to do that, but I don't know whether it's true. > If there are nasty cases like that, maybe what we have to do is allow > plpythonu/plpython2u functions to be dumped and reloaded into a > python-3-only install, but refuse to execute them until they've > been converted. > Unfortunately, I think there are cases like that. The shift to Unicode as the default string means that some functions that used to return a `str` now return a `bytes` (I know of this in the hashlib and base64 modules, but probably also in URL request data and others), and to use a `bytes` in string manipulation you have to first explicitly convert it to some string encoding. So things like a function that wraps around a python crypto library would be the exact places where those was-str-now-bytes functions would be used.
Re: Resolving the python 2 -> python 3 mess
Jesse Zhang writes: > I really like the "stub .so" idea, but feel pretty uncomfortable for the > "transparent" upgrade. Response inlined. Fair enough, but ... >> 2. On platforms where Python 2.x is no longer supported, transparently >> map plpythonu and plpython2u to plpython3u. "Transparent" meaning that >> dump/reload or pg_upgrade of existing plpythonu/plpython2u functions >> will work, but when you run them, what you actually get is Python 3.x. > It's fair enough that plpythonu changes its meaning, people who really > want the stability should explicitly use plpython2u. Yeah, but then what do you want to do with functions declared plpython2u? Have them fail even if they'd work fine under Python 3? Doesn't really seem like that's helping anyone. >> A possible gotcha in this approach is if there are any python 2/3 >> incompatibilities that would not manifest as syntax errors or >> obvious runtime errors, but would allow old code to execute and >> silently do the wrong thing. > "True division", one of the very first (2011, awww) few breaking changes > introduced in Python 3 [1], comes to mind. While it's not the worst > incompatibilities between Python 2 and 3, it's bad enough to give pause > to the notion that a successful parsing implies successful conversion. Hm. I agree that's kind of nasty, because 2to3 doesn't fix it AFAICT (and, likely, there is no way to do so that doesn't include solving the halting problem). However, it's not clear to me why forcing users to do a conversion is going to help them any with that, precisely because the automated conversion won't fix it. They're going to have to find such issues the hard way whenever they move to Python 3, no matter what we do. regards, tom lane
Re: Resolving the python 2 -> python 3 mess
Hi Tom, I really like the "stub .so" idea, but feel pretty uncomfortable for the "transparent" upgrade. Response inlined. On Mon, Feb 17, 2020 at 8:49 AM Tom Lane wrote: > > 2. On platforms where Python 2.x is no longer supported, transparently > map plpythonu and plpython2u to plpython3u. "Transparent" meaning that > dump/reload or pg_upgrade of existing plpythonu/plpython2u functions > will work, but when you run them, what you actually get is Python 3.x. It's fair enough that plpythonu changes its meaning, people who really want the stability should explicitly use plpython2u. > > For existing functions that don't use any obsolete Python syntax > (which one would hope is a pretty large percentage), this is a > zero-effort conversion for users. If a function does use obsolete > constructs, it will get a parse failure when executed, and the user > will have to update it to Python 3 syntax. I propose that we make > that case reasonably painless by providing the conversion script > I posted in [3] (or another one if somebody's got a better one), > bundled as a separately-installable extension. > > A possible gotcha in this approach is if there are any python 2/3 > incompatibilities that would not manifest as syntax errors or > obvious runtime errors, but would allow old code to execute and > silently do the wrong thing. One would hope that the Python crowd > weren't dumb enough to do that, but I don't know whether it's true. > If there are nasty cases like that, maybe what we have to do is allow > plpythonu/plpython2u functions to be dumped and reloaded into a > python-3-only install, but refuse to execute them until they've > been converted. "True division", one of the very first (2011, awww) few breaking changes introduced in Python 3 [1], comes to mind. While it's not the worst incompatibilities between Python 2 and 3, it's bad enough to give pause to the notion that a successful parsing implies successful conversion. [1] https://www.python.org/dev/peps/pep-0238/ Cheers, Jesse
Resolving the python 2 -> python 3 mess
We've had multiple previous discussions of $SUBJECT (eg [1][2]), without any resolution of what to do exactly. Thinking about this some more, I had an idea that I don't think has been discussed. To wit: 1. On platforms where Python 2.x is still supported, recommend that packagers continue to build both plpython2 and plpython3, same as now. 2. On platforms where Python 2.x is no longer supported, transparently map plpythonu and plpython2u to plpython3u. "Transparent" meaning that dump/reload or pg_upgrade of existing plpythonu/plpython2u functions will work, but when you run them, what you actually get is Python 3.x. For existing functions that don't use any obsolete Python syntax (which one would hope is a pretty large percentage), this is a zero-effort conversion for users. If a function does use obsolete constructs, it will get a parse failure when executed, and the user will have to update it to Python 3 syntax. I propose that we make that case reasonably painless by providing the conversion script I posted in [3] (or another one if somebody's got a better one), bundled as a separately-installable extension. A possible gotcha in this approach is if there are any python 2/3 incompatibilities that would not manifest as syntax errors or obvious runtime errors, but would allow old code to execute and silently do the wrong thing. One would hope that the Python crowd weren't dumb enough to do that, but I don't know whether it's true. If there are nasty cases like that, maybe what we have to do is allow plpythonu/plpython2u functions to be dumped and reloaded into a python-3-only install, but refuse to execute them until they've been converted. In either case, to allow dump/reload or pg_upgrade to work without ugly hacks, what we need to do is provide a stub version of plpython2.so. (The extension definitions that sit on top of it then don't need to change.) The stub would either redirect calls to plpython3.so if we prefer that approach, or throw errors if we prefer that approach. I envision adding a configure option that enables build and install of this stub library while doing a plpython3 build; packagers not planning to build a "real" plpython2 should ask for the stub instead. The end result given the first approach is that "plpythonu" and "plpython2u" and "plpython3u" all work and mean the same thing. Over some long time period we might want to deprecate and remove the "plpython2u" alias, but there would be no hurry about it. The work involved in making this happen seems fairly minimal, and practical to get done in time for PG 13. Perhaps there'd even be a case for back-patching it, though I'm not going to advocate for that here. Thoughts? regards, tom lane [1] https://www.postgresql.org/message-id/flat/5351890.TdMePpdHBD%40nb.usersys.redhat.com [2] https://www.postgresql.org/message-id/flat/CAKmB1PGDAy9mXxSTqUchYEi4iJAA6NKVj4P5BtAzvQ9wSDUwJw%40mail.gmail.com [3] https://www.postgresql.org/message-id/11546.1566584867%40sss.pgh.pa.us
