Re: [GENERAL] New user: Windows, Postgresql, Python
[EMAIL PROTECTED] (Marco Colombo) writes:
It seems python documentation is plain wrong, or I'm not able to
read it at all:
http://docs.python.org/ref/physical.html
A physical line ends in whatever the current platform's convention is for
terminating lines. On Unix, this is the ASCII LF (linefeed) character. On
Windows, it is the ASCII sequence CR LF (return followed by linefeed). On
Macintosh, it is the ASCII CR (return) character.
This is the language _reference_ manual, btw. I'm very surprised to hear
python on windows is so broken.
I believe this is wrong in two ways - first, it hasn't been updated
to cater for the recent Universal Newline support, and second, it
applies only to Python source code files (embedded code using the C
APIs should pass code using C newline conventions, ie \n characters,
as we have confirmed). I've submitted a Python bug report (SF ref
1167922) against the documentation.
I've suggested updated wording for this section as follows:
A physical line is a sequence of characters terminated by an end-of-line
sequence. In source files, any of the standard platform line
termination sequences can be used - the \UNIX form using \ASCII{} LF
(linefeed), the Windows form using the \ASCII{} sequence CR LF (return
followed by linefeed), or the Macintosh form using the \ASCII{} CR
(return) character. All of these forms can be used equally, regardless
of platform.
When embedding Python, source code strings should be passed to Python
APIs using the standard C conventions for newline characters (the
\code{\e n} character, representing \ASCII{} LF, is the line
terminator).
Is that clearer?
Paul.
--
Once the game is over, the King and the pawn go back in the same box. --
Italian Proverb
---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match
Re: [GENERAL] New user: Windows, Postgresql, Python
Michael Fuhr wrote: On Tue, Mar 15, 2005 at 10:46:09PM +, Paul Moore wrote: The long and short of it is that I believe you just use \n to delimit lines on Windows, just like anywhere else. Many thanks -- your test results contain the info we've been seeking. Thanks a lot Paul. Micheal, you were right. It seems python documentation is plain wrong, or I'm not able to read it at all: http://docs.python.org/ref/physical.html A physical line ends in whatever the current platform's convention is for terminating lines. On Unix, this is the ASCII LF (linefeed) character. On Windows, it is the ASCII sequence CR LF (return followed by linefeed). On Macintosh, it is the ASCII CR (return) character. This is the language _reference_ manual, btw. I'm very surprised to hear python on windows is so broken. Anyway, that makes life simpler for us. plpython programs are \n separated, no matter what platform the server runs on. Client applications just need to conply, which is what I suggested some time ago. I'm glad to hear there's nothing to change on the server side. .TM. -- / / / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _/ _/ _/ [EMAIL PROTECTED] ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [GENERAL] New user: Windows, Postgresql, Python
[EMAIL PROTECTED] (Marco Colombo) writes: No I wasn't sure and I actually was wrong. I've never programmed under Windows. I've just learned something. Indeed, the Windows C runtime translates CRLF to \n on input, and \n to CRLF on output, for files in text mode. Unix programmers tend not to be aware of the distinction between text and binary modes (it's actually in standard C) as it makes no difference on Unix. But it does on Windows (and possibly other platforms). offtopic Ironically, at the lowest level, Windows behaves just like Unix (files are pure byte streams) - it's only in the C runtime and application code that CRLF issues arise, and that's a backward-compatibility hack dating back to the days of MS-DOS. /offtopic Apparently, as far as Python is concerned, the platform presents \n at C level, so it makes sense for PyRun_SimpleString() to expect \n as line terminator. Still I don't understand when the lexxer would use \r\n as pysical line ending on Windows, but I can live with it. :-) Internally, Python uses C string semantics, where \n represents a newline. Recent versions of Python have universal newline support, which in the broadest sense attempts to be forgiving over line endings, and treat LF, CRLF, and even bare CR, as line endings. I don't know exactly where it applies, though, so I believe the most sensible approach is to always use \n (LF) in strings passed to Python APIs. This is essentially the be conservative in what you send philosophy. Paul. -- A little inaccuracy sometimes saves tons of explanation -- Saki ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [GENERAL] New user: Windows, Postgresql, Python
On Thursday 17 March 2005 23:17, Paul Moore wrote: offtopic Ironically, at the lowest level, Windows behaves just like Unix (files are pure byte streams) - it's only in the C runtime and application code that CRLF issues arise, and that's a backward-compatibility hack dating back to the days of MS-DOS. /offtopic Even more offtopic: Actually, the CR/LF pair dates back to the ancient teletype writers, which needed one character for the right-to-left movement of the paper carriage (hence the literal meaning of Carriage Return), and one for the vertical movement. I believe it was Tom Swan who, in his Programming Turbo Pascal from the eighties, said something to the effect that this is not only a case of the tail wagging the dog, but a tail that keeps on wagging twenty years after the dog has rolled over and died. Sorry-for-spinning-of-on-a-tangent-ly yours - -- Leif Biberg Kristensen http://solumslekt.org/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [GENERAL] New user: Windows, Postgresql, Python
On Wed, Mar 16, 2005 at 01:46:23PM +0100, Marco Colombo wrote: It seems python documentation is plain wrong, or I'm not able to read it at all: http://docs.python.org/ref/physical.html A physical line ends in whatever the current platform's convention is for terminating lines. On Unix, this is the ASCII LF (linefeed) character. On Windows, it is the ASCII sequence CR LF (return followed by linefeed). On Macintosh, it is the ASCII CR (return) character. Perhaps the Python documentation could use some clarification about when the platform's convention is required and when it isn't. The Embedding Python documentation shows embedded code with lines ending in \n and without saying anything about requiring the platform's convention: http://docs.python.org/ext/high-level-embedding.html This is the language _reference_ manual, btw. I'm very surprised to hear python on windows is so broken. Anyway, that makes life simpler for us. plpython programs are \n separated, no matter what platform the server runs on. That the behavior makes life simpler is an argument against it being broken (although it would be even less broken if it were more flexible about what line endings it allows). A detailed response would be getting off-topic for PostgreSQL, but I'll stand by what I said earlier: I would find it bizarre if embedded Python code had to use different line endings on different platforms. That would mean the programmer couldn't simply do this: PyRun_SimpleString(x = 1\n print x\n); Instead, the programmer would have to do a compile-time or run-time check and build the string in a platform-dependent manner. What problem would the language be solving by requiring that? -- Michael Fuhr http://www.fuhr.org/~mfuhr/ ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [GENERAL] New user: Windows, Postgresql, Python
On Wed, 16 Mar 2005, Michael Fuhr wrote: On Wed, Mar 16, 2005 at 01:46:23PM +0100, Marco Colombo wrote: It seems python documentation is plain wrong, or I'm not able to read it at all: http://docs.python.org/ref/physical.html A physical line ends in whatever the current platform's convention is for terminating lines. On Unix, this is the ASCII LF (linefeed) character. On Windows, it is the ASCII sequence CR LF (return followed by linefeed). On Macintosh, it is the ASCII CR (return) character. Perhaps the Python documentation could use some clarification about when the platform's convention is required and when it isn't. The Embedding Python documentation shows embedded code with lines ending in \n and without saying anything about requiring the platform's convention: http://docs.python.org/ext/high-level-embedding.html This is the language _reference_ manual, btw. I'm very surprised to hear python on windows is so broken. Anyway, that makes life simpler for us. plpython programs are \n separated, no matter what platform the server runs on. That the behavior makes life simpler is an argument against it being broken (although it would be even less broken if it were more flexible about what line endings it allows). broken == 'not conforming to the specifications or the documentation' The fact it helps us is just a side effect. A detailed response would be getting off-topic for PostgreSQL, but I'll stand by what I said earlier: I would find it bizarre if embedded Python code had to use different line endings on different platforms. That would mean the programmer couldn't simply do this: PyRun_SimpleString(x = 1\n print x\n); Instead, the programmer would have to do a compile-time or run-time check and build the string in a platform-dependent manner. What problem would the language be solving by requiring that? This one: aprogram = x = 1\nprint x\n; printf(aprogram); PyRun_SimpleString(aprogram); See? THIS program requires compile-time or run-time checks. You can't run it on Windows, or Mac: it'll write garbage to the screen (something that looks like garbage, that is). Make it more general: aprogram = get_program_from_somewhere(); PyRun_SimpleString(aprogram); write_program_to_somefile_possibly_stdout(aprogram); What if get_program_from_somewhere() reads user input? On Windows lines will be \r\n separated. Now, should this program make platform checks? Why not simply read a file (or stdin) in text mode, and pass the result to PyRun_SimpleString()? The same applies to output, of course. Now something strikes me... in his tests, Paul tried my program and the output looks identical to Linux. Now... I was expecting program1 (the one with just \n) do display badly under Windows. Am I missing something? Does C runtime support in Windows convert \n into \r\n automatically in printf()? If so, I'm on the wrong track. It may do the same with scanf() and other stdio functions. I must say I wasn't expecting my program to run just fine, with all those \n I used in it. Staring from printf( Initialized.\n); Paul can you please tell me which compiler you used under Windows to complile my program and if you used some weird compiling options? TIA. .TM. -- / / / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _/ _/ _/ [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [GENERAL] New user: Windows, Postgresql, Python
On Wed, Mar 16, 2005 at 04:17:51PM +0100, Marco Colombo wrote: aprogram = x = 1\nprint x\n; printf(aprogram); PyRun_SimpleString(aprogram); See? THIS program requires compile-time or run-time checks. You can't run it on Windows, or Mac: it'll write garbage to the screen (something that looks like garbage, that is). Are you sure about that? It's been forever since I programmed in a Microsoft environment, but as I recall, I/O streams opened in text mode do automatic translations between \n and \r\n. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/HTML/_crt_fopen.2c_._wfopen.asp Also, in text mode, carriage return-linefeed combinations are translated into single linefeeds on input, and linefeed characters are translated to carriage return-linefeed combinations on output. I didn't look up Mac behavior but I'd be surprised if it didn't offer the same text mode and binary mode behaviors. It's annoying that these platforms use different line endings, but at least their implementations of standard C libraries offer a way to hide that difference from the programmer. Now something strikes me... in his tests, Paul tried my program and the output looks identical to Linux. Now... I was expecting program1 (the one with just \n) do display badly under Windows. Am I missing something? Does C runtime support in Windows convert \n into \r\n automatically in printf()? If so, I'm on the wrong track. It may do the same with scanf() and other stdio functions. I think that's exactly what happens with I/O streams in text mode. -- Michael Fuhr http://www.fuhr.org/~mfuhr/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [GENERAL] New user: Windows, Postgresql, Python
On Wed, 16 Mar 2005, Michael Fuhr wrote: On Wed, Mar 16, 2005 at 04:17:51PM +0100, Marco Colombo wrote: aprogram = x = 1\nprint x\n; printf(aprogram); PyRun_SimpleString(aprogram); See? THIS program requires compile-time or run-time checks. You can't run it on Windows, or Mac: it'll write garbage to the screen (something that looks like garbage, that is). Are you sure about that? It's been forever since I programmed in a Microsoft environment, but as I recall, I/O streams opened in text mode do automatic translations between \n and \r\n. No I wasn't sure and I actually was wrong. I've never programmed under Windows. I've just learned something. Apparently, as far as Python is concerned, the platform presents \n at C level, so it makes sense for PyRun_SimpleString() to expect \n as line terminator. Still I don't understand when the lexxer would use \r\n as pysical line ending on Windows, but I can live with it. :-) It seems that any client application under Windows is likely to use only \n-delimited text, as long as it uses stdio functions and text mode. Problems arise when it gets text from some other source. But since at C level text is expected to be \n-delimited, the application should take care of the conversion as soon as it receives the data. I think that if we want to be conservative, any input that is supposed to be treated (actively) as text by the server, should be \n-delimited. That includes any function source. I'm against to any on-the-fly conversion, now. I don't like the idea of PostgreSQL accepting input in one form (\r\n) and providing output in a different form (\n). Also think of a function definition with mixed \r\n and \n lines: we'd have no way to reconstruct the original input. I think we should just state that text used for function definitions is \n-delimited. Some languages may accept \r\n as well, but that's undocumented side effect, and bad practice. Now that I learned that C programs on Windows are expected to handle \n-delimited text, I can't think of any reason why an application should send \r\n-delimited text via libpq as a function definition, unless the programmer forgot to perform the standard \r\n to \n conversion somewhere. .TM. -- / / / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _/ _/ _/ [EMAIL PROTECTED] ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [GENERAL] New user: Windows, Postgresql, Python
[EMAIL PROTECTED] (Magnus Hagander) writes: I suppose my first (lazy) question is, is there a Python 2.4 compatible plpython.dll available anywhere? Alternatively, is there a way I can build one for myself? I'm happy enough doing my own build (I have mingw and msys available), but I'd rather not build the whole of postgresql if possible, just for the sake of one DLL Not that I know of. IFF the libraries export the same entrypoints without changing things, you could try just copying python24.dll to python23.dll. I don't know how the Python guys are with binary compatibility, though. Might be worth a shot. As per my earlier posting, I actually found that building postgresql wasn't at all hard. Once I'd built with Python 2.4 support, I had a compatible plpython.dll I could just copy in. I'm not sure renaming the Python DLL would have worked - it's definitely frowned on... On a different note, can't you have both python 2.3 *and* 2.4 on the asme system? Considering they put the version number in the filename, it seems this should be possible? I could, but I try to avoid this, as it involves double installs of any extensions I want to use, or incompatible environments. More laziness on my part, really :-) Thanks for the suggestions, Paul. PS Thanks to the developers who made building postgresql on Windows such an easy job! I was very impressed - I genuinely didn't think that building such a large piece of software would be so hassle-free. -- Never keep up with the Joneses. Drag them down to your level. -- Quentin Crisp ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [GENERAL] New user: Windows, Postgresql, Python
On Tue, Mar 15, 2005 at 07:05:22PM +, Paul Moore wrote: As per my earlier posting, I actually found that building postgresql wasn't at all hard. Once I'd built with Python 2.4 support, I had a compatible plpython.dll I could just copy in. Pardon the interruption, but do you have a PostgreSQL server with PL/Python running on Windows? Have you been following the plpython function problem workaround thread? http://archives.postgresql.org/pgsql-general/2005-03/msg00599.php We (the thread participants) could use somebody with a Windows server to do some testing. Specifically, we're wondering if Python on Windows requires embedded Python code to have CRLF (\r\n) as a line ending, or if it requires (or at least permits) LF (\n) only. If you're able to help, could you could post the results of the following? CREATE FUNCTION pytest_lf() RETURNS integer AS 'x = 1\nreturn x\n' LANGUAGE plpythonu; CREATE FUNCTION pytest_crlf() RETURNS integer AS 'x = 1\r\nreturn x\r\n' LANGUAGE plpythonu; SELECT pytest_lf(); SELECT pytest_crlf(); With PostgreSQL 8.0.1, Python 2.4.1c1, and Solaris 9, I get this: test=# SELECT pytest_lf(); pytest_lf --- 1 (1 row) test=# SELECT pytest_crlf(); ERROR: plpython: could not compile function pytest_crlf DETAIL: exceptions.SyntaxError: invalid syntax (line 2) If you have the ability to compile standalone C programs with embedded Python, we'd also be interested in seeing what happens if you run the programs in the following messages: http://archives.postgresql.org/pgsql-general/2005-01/msg00876.php http://archives.postgresql.org/pgsql-general/2005-03/msg00630.php Any test results or comments you can provide would be appreciated. Thanks. -- Michael Fuhr http://www.fuhr.org/~mfuhr/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [GENERAL] New user: Windows, Postgresql, Python
[EMAIL PROTECTED] (Michael Fuhr) writes: We (the thread participants) could use somebody with a Windows server to do some testing. Glad to help... This is with postgresql 8.0.1, Python 2.4. Specifically, we're wondering if Python on Windows requires embedded Python code to have CRLF (\r\n) as a line ending, or if it requires (or at least permits) LF (\n) only. If you're able to help, could you could post the results of the following? CREATE FUNCTION pytest_lf() RETURNS integer AS 'x = 1\nreturn x\n' LANGUAGE plpythonu; CREATE FUNCTION pytest_crlf() RETURNS integer AS 'x = 1\r\nreturn x\r\n' LANGUAGE plpythonu; SELECT pytest_lf(); SELECT pytest_crlf(); With PostgreSQL 8.0.1, Python 2.4.1c1, and Solaris 9, I get this: test=# SELECT pytest_lf(); pytest_lf --- 1 (1 row) test=# SELECT pytest_crlf(); ERROR: plpython: could not compile function pytest_crlf DETAIL: exceptions.SyntaxError: invalid syntax (line 2) I get exactly the same results. If you have the ability to compile standalone C programs with embedded Python, we'd also be interested in seeing what happens if you run the programs in the following messages: http://archives.postgresql.org/pgsql-general/2005-01/msg00876.php I get: test1 What hath Guido wrought? http://archives.postgresql.org/pgsql-general/2005-03/msg00630.php I get: test2 Initialized. Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] running: print 1 print 2 1 2 end running: print 1 print 2 File string, line 1 print 1 ^ SyntaxError: invalid syntax end Finalized. I don't know if this helps? It seems reasonable to me - as far as Python C code is concerned, code strings should be \n-separated, just like in Unix. The only place CRLF is applicable is in code read from files, where the C runtime converts it to \n-delimited before the Python APIs see it (as far as I understand it, which isn't very far...) The long and short of it is that I believe you just use \n to delimit lines on Windows, just like anywhere else. Regards, Paul. -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [GENERAL] New user: Windows, Postgresql, Python
On Tue, Mar 15, 2005 at 10:46:09PM +, Paul Moore wrote: The long and short of it is that I believe you just use \n to delimit lines on Windows, just like anywhere else. Many thanks -- your test results contain the info we've been seeking. -- Michael Fuhr http://www.fuhr.org/~mfuhr/ ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [GENERAL] New user: Windows, Postgresql, Python
Hi, I'm just starting to look at Postgresql. My platform (for better or worse) is Windows, and I'm quite interested in the pl/python support. However, when I run the binary installer, it is not offered to me as an option (it's there, but greyed out). The plpython.dll file is installed, however. When I check, it looks like plpython.dll is linked against Python 2.3. I have Python 2.4 installed on my PC, and I don't really want to downgrade. I suppose my first (lazy) question is, is there a Python 2.4 compatible plpython.dll available anywhere? Alternatively, is there a way I can build one for myself? I'm happy enough doing my own build (I have mingw and msys available), but I'd rather not build the whole of postgresql if possible, just for the sake of one DLL Not that I know of. IFF the libraries export the same entrypoints without changing things, you could try just copying python24.dll to python23.dll. I don't know how the Python guys are with binary compatibility, though. Might be worth a shot. On a different note, can't you have both python 2.3 *and* 2.4 on the asme system? Considering they put the version number in the filename, it seems this should be possible? //Magnus ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[GENERAL] New user: Windows, Postgresql, Python
Hi, I'm just starting to look at Postgresql. My platform (for better or worse) is Windows, and I'm quite interested in the pl/python support. However, when I run the binary installer, it is not offered to me as an option (it's there, but greyed out). The plpython.dll file is installed, however. When I check, it looks like plpython.dll is linked against Python 2.3. I have Python 2.4 installed on my PC, and I don't really want to downgrade. I suppose my first (lazy) question is, is there a Python 2.4 compatible plpython.dll available anywhere? Alternatively, is there a way I can build one for myself? I'm happy enough doing my own build (I have mingw and msys available), but I'd rather not build the whole of postgresql if possible, just for the sake of one DLL Thanks in advance, Paul. -- Bother, said the Borg, We've assimilated Pooh. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [GENERAL] New user: Windows, Postgresql, Python
Paul Moore [EMAIL PROTECTED] writes: I suppose my first (lazy) question is, is there a Python 2.4 compatible plpython.dll available anywhere? Alternatively, is there a way I can build one for myself? I'm happy enough doing my own build (I have mingw and msys available), but I'd rather not build the whole of postgresql if possible, just for the sake of one DLL Actually, I had a go and was surprised to find that the build was pretty simple. Once I'd got a build, copying the plpython DLL (which is built with a different name, AFAICT) over the one installed by the binary installer seems to work fine. Is that OK, or could I hit problems later? Paul. -- Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something. -- Robert Heinlein ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
