[Python-Dev] Problem with pydoc script in PEP 405 venvs

2013-06-17 Thread Vinay Sajip 
Issue #18224 (http://bugs.python.org/issue18224) highlights a problem on
Windows with the pydoc script provided with venvs created by pyvenv. On
POSIX, the script is named pydoc and causes no problems: on Windows, it is
called pydoc.py and this causes problems because it shadows the stdlib pydoc
module.

Possible solutions:

1. Remove the pydoc script altogether from created venvs, on Windows but
also on POSIX (for consistency).
2. Rename the pydoc script on both Windows and POSIX (e.g. to pydocs.py and
pydocs respectively).
3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe
launcher pydoc.exe adjacent to it (which is how setuptools and distlib
handle installed scripts).

The first two approaches are backwards-incompatible, while the third is less
likely to lead to breakage, but involves adding a Windows script launcher to
Python. While this is a bigger change, I think any built-in Python installer
functionality should include such a launcher (as setuptools and distlib do).
Still, that's probably a discussion for another day.

Does anyone have any comments? Approach #2 seems the most appropriate. I
assume it would be reasonable to implement this in both 3.3 and 3.4, as it's
not a change in core Python APIs.

In the absence of adverse feedback here, I propose to implement approach #2
on both 3.3 and 3.4.

Regards,

Vinay Sajip

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Writing Extensions for Python 3 in Windows

2013-06-17 Thread Aditya Avinash 
Hi. This is the last place where I want to ask a question. I have searched
for lots of tutorials and documentation on the web but, didn't find a
decent one to develop extensions for Python 3 using a custom compiler
(mingw32, nvcc). Please help me.
PS: Don't point me to Python Documentation. It is not good for beginners.
It doesn't elaborate about calls and implementation.

-- 
Aditya Avinash Atluri
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Walter Dörwald 

On 14.06.13 23:03, PJ Eby wrote:

On Fri, Jun 14, 2013 at 2:11 PM, Ron Adam  wrote:



On 06/14/2013 10:36 AM, Guido van Rossum wrote:


Not a bug. The same is done for file input -- CRLF is changed to LF before
tokenizing.




Should this be the same?


python3 -c 'print(bytes("""\r\n""", "utf8"))'
b'\r\n'



eval('print(bytes("""\r\n""", "utf8"))')

b'\n'


No, but:

eval(r'print(bytes("""\r\n""", "utf8"))')

should be.  (And is.)

What I believe you and Walter are missing is that the \r\n in the eval
strings are converted early if you don't make the enclosing string
raw.  So what you're eval-ing is not what you think you are eval-ing,
hence the confusion.


I expected that eval()ing a string that contains the characters

   U+0027: APOSTROPHE
   U+0027: APOSTROPHE
   U+0027: APOSTROPHE
   U+000D: CR
   U+000A: LR
   U+0027: APOSTROPHE
   U+0027: APOSTROPHE
   U+0027: APOSTROPHE

to return a string containing the characters:

   U+000D: CR
   U+000A: LR

Making the string raw, of course turns it into:

   U+0027: APOSTROPHE
   U+0027: APOSTROPHE
   U+0027: APOSTROPHE
   U+005C: REVERSE SOLIDUS
   U+0072: LATIN SMALL LETTER R
   U+005C: REVERSE SOLIDUS
   U+006E: LATIN SMALL LETTER N
   U+0027: APOSTROPHE
   U+0027: APOSTROPHE
   U+0027: APOSTROPHE

and eval()ing that does indeed give "\r\n" as expected.

Hmm, it seems that codecs.unicode_escape_decode() does what I want:

>>> codecs.unicode_escape_decode("\r\n\\r\\n\\x0d\\x0a\\u000d\\u000a")
('\r\n\r\n\r\n\r\n', 26)

Servus,
   Walter

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with pydoc script in PEP 405 venvs

2013-06-17 Thread Richard Oudkerk 

On 17/06/2013 5:27pm, Vinay Sajip wrote:

3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe
launcher pydoc.exe adjacent to it (which is how setuptools and distlib
handle installed scripts).


Can't a batch file pydoc.bat be used?

--
Richard

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Writing Extensions for Python 3 in Windows

2013-06-17 Thread Brett Cannon 
So the best place to ask this question is python-list and not python-dev as
this is about developing *for* Python and not *with* Python.


On Mon, Jun 17, 2013 at 12:42 PM, Aditya Avinash  wrote:

> Hi. This is the last place where I want to ask a question. I have searched
> for lots of tutorials and documentation on the web but, didn't find a
> decent one to develop extensions for Python 3 using a custom compiler
> (mingw32, nvcc). Please help me.
> PS: Don't point me to Python Documentation. It is not good for beginners.
> It doesn't elaborate about calls and implementation.
>
> --
> Aditya Avinash Atluri
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Walter Dörwald 

On 17.06.13 19:04, Walter Dörwald wrote:


Hmm, it seems that codecs.unicode_escape_decode() does what I want:

 >>> codecs.unicode_escape_decode("\r\n\\r\\n\\x0d\\x0a\\u000d\\u000a")
('\r\n\r\n\r\n\r\n', 26)


Hmm, no it doesn't:

>>> codecs.unicode_escape_decode("\u1234")
('á\x88´', 3)

Servus,
   Walter

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Guido van Rossum 
On Mon, Jun 17, 2013 at 10:04 AM, Walter Dörwald  wrote:
> I expected that eval()ing a string that contains the characters
>
>U+0027: APOSTROPHE
>U+0027: APOSTROPHE
>U+0027: APOSTROPHE
>U+000D: CR
>U+000A: LR
>U+0027: APOSTROPHE
>U+0027: APOSTROPHE
>U+0027: APOSTROPHE
>
> to return a string containing the characters:
>
>U+000D: CR
>U+000A: LR

No. Executing a file containing those exact characters produces a
string containing only '\n' and exec/eval is meant to behave the same
way. The string may not have originated from a file, so the universal
newlines behavior of the io module is irrelevant here -- the parser
must implement its own equivalent processing, and it does.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with pydoc script in PEP 405 venvs

2013-06-17 Thread Vinay Sajip 
Richard Oudkerk  gmail.com> writes:

> 
> Can't a batch file pydoc.bat be used?
> 

I generally find .bat files so limiting that I never thought of that. In
this case, it makes perfect sense to use one. Thanks!

@echo off
__VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())"

should do it.

Regards,

Vinay Sajip

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with pydoc script in PEP 405 venvs

2013-06-17 Thread R. David Murray 
On Mon, 17 Jun 2013 16:27:45 -, Vinay Sajip  wrote:
> Issue #18224 (http://bugs.python.org/issue18224) highlights a problem on
> Windows with the pydoc script provided with venvs created by pyvenv. On
> POSIX, the script is named pydoc and causes no problems: on Windows, it is
> called pydoc.py and this causes problems because it shadows the stdlib pydoc
> module.
> 
> Possible solutions:
> 
> 1. Remove the pydoc script altogether from created venvs, on Windows but
> also on POSIX (for consistency).
> 2. Rename the pydoc script on both Windows and POSIX (e.g. to pydocs.py and
> pydocs respectively).
> 3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe
> launcher pydoc.exe adjacent to it (which is how setuptools and distlib
> handle installed scripts).
> 
> The first two approaches are backwards-incompatible, while the third is less
> likely to lead to breakage, but involves adding a Windows script launcher to
> Python. While this is a bigger change, I think any built-in Python installer
> functionality should include such a launcher (as setuptools and distlib do).
> Still, that's probably a discussion for another day.
> 
> Does anyone have any comments? Approach #2 seems the most appropriate. I
> assume it would be reasonable to implement this in both 3.3 and 3.4, as it's
> not a change in core Python APIs.
> 
> In the absence of adverse feedback here, I propose to implement approach #2
> on both 3.3 and 3.4.

I don't think renaming pydoc to pydocs is appropriate on posix.  Too many
people likely have 'pydoc' in their fingers and brains as the command name.

The .bat file suggestion seems better, IMO.

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with pydoc script in PEP 405 venvs

2013-06-17 Thread Oleg Broytman 
On Mon, Jun 17, 2013 at 06:37:49PM +, Vinay Sajip  
wrote:
> @echo off
> __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())"

   I think you want to pass command line arguments:

@echo off
__VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" %1 %2 %3 %4 %5 %6 
%7 %8 %9

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/[email protected]
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with pydoc script in PEP 405 venvs

2013-06-17 Thread Vinay Sajip 
Oleg Broytman  phdru.name> writes:

> 
> On Mon, Jun 17, 2013 at 06:37:49PM +, Vinay Sajip 
yahoo.co.uk> wrote:
> >   echo off
> > __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())"
> 
>I think you want to pass command line arguments:
> 
>   echo off
> __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" %1 %2 %3 %4
%5 %6 %7 %8 %9
> 

That was off-the-cuff, what I actually implemented (in my sandbox repo) was

@echo off
__VENV_PYTHON__ -c "import sys, pydoc; sys.argv[0] = 'pydoc';
sys.exit(pydoc.cli())" %*

Regards,

Vinay Sajip


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Ron Adam 



On 06/17/2013 12:04 PM, Walter Dörwald wrote:

Making the string raw, of course turns it into:

U+0027: APOSTROPHE
U+0027: APOSTROPHE
U+0027: APOSTROPHE
U+005C: REVERSE SOLIDUS
U+0072: LATIN SMALL LETTER R
U+005C: REVERSE SOLIDUS
U+006E: LATIN SMALL LETTER N
U+0027: APOSTROPHE
U+0027: APOSTROPHE
U+0027: APOSTROPHE

and eval()ing that does indeed give "\r\n" as expected.


You can also escape the reverse slashes in a regular string to get the same 
result.



>>> s1 = "'''\\r\\n'''"
>>> list(s1)
["'", "'", "'", '\\', 'r', '\\', 'n', "'", "'", "'"]

>>> s2 = eval(s1)
>>> list(s2)
['\r', '\n']

>>> s3 = "'''%s'''" % s2
>>> list(s3)
["'", "'", "'", '\r', '\n', "'", "'", "'"]

>>> s4 = eval(s3)
>>> list(s4)
['\n']


When a standard string literal used with eval, it's evaluated first to a 
string object in the same scope as the eval function is called from, then 
the eval function is called with that string object and it's evaluated 
again.  So it's really being parsed twice.  (That's the part that got me.)


The transformation between s1 and s2 is what Phillip is referring to, and
Guido is referring to the transformation from s2 to s4.   (s3 is needed to 
avoid the end of line error of evaluating a single quoted string with \n in 
it.)


When a sting literal is used directly with eval, it looks like it is 
evaluated from s1 to s4 in one step, but that isn't what is happening.


Cheers,  Ron


(ps: Was informed my posts where showing up twice.. hopefully I got that 
fixed now.)















___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Greg Ewing 

Guido van Rossum wrote:

No. Executing a file containing those exact characters produces a
string containing only '\n' and exec/eval is meant to behave the same
way. The string may not have originated from a file, so the universal
newlines behavior of the io module is irrelevant here -- the parser
must implement its own equivalent processing, and it does.


I'm still not convinced that this is necessary or desirable
behaviour. I can understand the parser doing this as a
workaround before we had universal newlines, but now that
we do, I'd expect any Python string to already have newlines
converted to their canonical representation, and that any CRs
it contains are meant to be there. The parser shouldn't need
to do newline translation a second time.

--
Greg
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Guido van Rossum 
On Mon, Jun 17, 2013 at 3:18 PM, Greg Ewing  wrote:
> Guido van Rossum wrote:
>>
>> No. Executing a file containing those exact characters produces a
>> string containing only '\n' and exec/eval is meant to behave the same
>> way. The string may not have originated from a file, so the universal
>> newlines behavior of the io module is irrelevant here -- the parser
>> must implement its own equivalent processing, and it does.

> I'm still not convinced that this is necessary or desirable
> behaviour. I can understand the parser doing this as a
> workaround before we had universal newlines, but now that
> we do, I'd expect any Python string to already have newlines
> converted to their canonical representation, and that any CRs
> it contains are meant to be there. The parser shouldn't need
> to do newline translation a second time.

There are other ways to get a string besides reading it from a file.

Anyway, I think that if you want a string literal that contains \r\n
as its line endings, you should use a syntactic solution, and the
syntax ought to be the same regardless of whether you are reading it
from a file or from a string literal. That syntactic solution is very
clear:

"""line one\r
line two\r
line three\r
"""

This works everywhere.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Benjamin Peterson 
2013/6/17 Greg Ewing :
> Guido van Rossum wrote:
>>
>> No. Executing a file containing those exact characters produces a
>> string containing only '\n' and exec/eval is meant to behave the same
>> way. The string may not have originated from a file, so the universal
>> newlines behavior of the io module is irrelevant here -- the parser
>> must implement its own equivalent processing, and it does.
>
>
> I'm still not convinced that this is necessary or desirable
> behaviour. I can understand the parser doing this as a
> workaround before we had universal newlines, but now that
> we do, I'd expect any Python string to already have newlines
> converted to their canonical representation, and that any CRs
> it contains are meant to be there. The parser shouldn't need
> to do newline translation a second time.

It used to be that way until 2.7. People like to do things like

with open("myfile.py", "rb") as fp:
exec fp.read() in ns

which used to fail with CRLF newlines because binary mode doesn't have
them. I think this is actually the correct way to execute Python
sources because the parser then handles the somewhat complicated
process of decoding Python source for you.



--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Guido van Rossum 
On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson  wrote:
> 2013/6/17 Greg Ewing :
>> Guido van Rossum wrote:
>>>
>>> No. Executing a file containing those exact characters produces a
>>> string containing only '\n' and exec/eval is meant to behave the same
>>> way. The string may not have originated from a file, so the universal
>>> newlines behavior of the io module is irrelevant here -- the parser
>>> must implement its own equivalent processing, and it does.
>>
>>
>> I'm still not convinced that this is necessary or desirable
>> behaviour. I can understand the parser doing this as a
>> workaround before we had universal newlines, but now that
>> we do, I'd expect any Python string to already have newlines
>> converted to their canonical representation, and that any CRs
>> it contains are meant to be there. The parser shouldn't need
>> to do newline translation a second time.
>
> It used to be that way until 2.7. People like to do things like
>
> with open("myfile.py", "rb") as fp:
> exec fp.read() in ns
>
> which used to fail with CRLF newlines because binary mode doesn't have
> them. I think this is actually the correct way to execute Python
> sources because the parser then handles the somewhat complicated
> process of decoding Python source for you.

What exactly does the parser handles better than the io module? Is it
just the coding cookies? I suppose that works as long as the file is
encoded using as ASCII superset like the Latin-N variants or UTF-8. It
would fail pretty badly if it was UTF-16 (and yes, that's an
abominable encoding for other reasons :-).
-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Benjamin Peterson 
2013/6/17 Guido van Rossum :
> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson  
> wrote:
>> 2013/6/17 Greg Ewing :
>>> Guido van Rossum wrote:

 No. Executing a file containing those exact characters produces a
 string containing only '\n' and exec/eval is meant to behave the same
 way. The string may not have originated from a file, so the universal
 newlines behavior of the io module is irrelevant here -- the parser
 must implement its own equivalent processing, and it does.
>>>
>>>
>>> I'm still not convinced that this is necessary or desirable
>>> behaviour. I can understand the parser doing this as a
>>> workaround before we had universal newlines, but now that
>>> we do, I'd expect any Python string to already have newlines
>>> converted to their canonical representation, and that any CRs
>>> it contains are meant to be there. The parser shouldn't need
>>> to do newline translation a second time.
>>
>> It used to be that way until 2.7. People like to do things like
>>
>> with open("myfile.py", "rb") as fp:
>> exec fp.read() in ns
>>
>> which used to fail with CRLF newlines because binary mode doesn't have
>> them. I think this is actually the correct way to execute Python
>> sources because the parser then handles the somewhat complicated
>> process of decoding Python source for you.
>
> What exactly does the parser handles better than the io module? Is it
> just the coding cookies? I suppose that works as long as the file is
> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
> would fail pretty badly if it was UTF-16 (and yes, that's an
> abominable encoding for other reasons :-).

The coding cookie is the main one. In fact, if you can't parse that,
you don't really know what encoding to open the file with at all.
There's also small things like BOM handling (you have to use the
utf-16-sig encoding with TextIO to get it removed) and defaulting to
UTF-8 (which the io module doesn't do) which is better left to the
parser.


--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Guido van Rossum 
On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson  wrote:
> 2013/6/17 Guido van Rossum :
>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson  
>> wrote:
>>> 2013/6/17 Greg Ewing :
 Guido van Rossum wrote:
>
> No. Executing a file containing those exact characters produces a
> string containing only '\n' and exec/eval is meant to behave the same
> way. The string may not have originated from a file, so the universal
> newlines behavior of the io module is irrelevant here -- the parser
> must implement its own equivalent processing, and it does.


 I'm still not convinced that this is necessary or desirable
 behaviour. I can understand the parser doing this as a
 workaround before we had universal newlines, but now that
 we do, I'd expect any Python string to already have newlines
 converted to their canonical representation, and that any CRs
 it contains are meant to be there. The parser shouldn't need
 to do newline translation a second time.
>>>
>>> It used to be that way until 2.7. People like to do things like
>>>
>>> with open("myfile.py", "rb") as fp:
>>> exec fp.read() in ns
>>>
>>> which used to fail with CRLF newlines because binary mode doesn't have
>>> them. I think this is actually the correct way to execute Python
>>> sources because the parser then handles the somewhat complicated
>>> process of decoding Python source for you.
>>
>> What exactly does the parser handles better than the io module? Is it
>> just the coding cookies? I suppose that works as long as the file is
>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
>> would fail pretty badly if it was UTF-16 (and yes, that's an
>> abominable encoding for other reasons :-).
>
> The coding cookie is the main one. In fact, if you can't parse that,
> you don't really know what encoding to open the file with at all.
> There's also small things like BOM handling (you have to use the
> utf-16-sig encoding with TextIO to get it removed) and defaulting to
> UTF-8 (which the io module doesn't do) which is better left to the
> parser.

Maybe there are some lessons here that the TextIO module could learn?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Victor Stinner 
It may be possible to implement parsing the codec cookie as a Python codec :-)

Victor

2013/6/18 Guido van Rossum :
> On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson  
> wrote:
>> 2013/6/17 Guido van Rossum :
>>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson  
>>> wrote:
 2013/6/17 Greg Ewing :
> Guido van Rossum wrote:
>>
>> No. Executing a file containing those exact characters produces a
>> string containing only '\n' and exec/eval is meant to behave the same
>> way. The string may not have originated from a file, so the universal
>> newlines behavior of the io module is irrelevant here -- the parser
>> must implement its own equivalent processing, and it does.
>
>
> I'm still not convinced that this is necessary or desirable
> behaviour. I can understand the parser doing this as a
> workaround before we had universal newlines, but now that
> we do, I'd expect any Python string to already have newlines
> converted to their canonical representation, and that any CRs
> it contains are meant to be there. The parser shouldn't need
> to do newline translation a second time.

 It used to be that way until 2.7. People like to do things like

 with open("myfile.py", "rb") as fp:
 exec fp.read() in ns

 which used to fail with CRLF newlines because binary mode doesn't have
 them. I think this is actually the correct way to execute Python
 sources because the parser then handles the somewhat complicated
 process of decoding Python source for you.
>>>
>>> What exactly does the parser handles better than the io module? Is it
>>> just the coding cookies? I suppose that works as long as the file is
>>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
>>> would fail pretty badly if it was UTF-16 (and yes, that's an
>>> abominable encoding for other reasons :-).
>>
>> The coding cookie is the main one. In fact, if you can't parse that,
>> you don't really know what encoding to open the file with at all.
>> There's also small things like BOM handling (you have to use the
>> utf-16-sig encoding with TextIO to get it removed) and defaulting to
>> UTF-8 (which the io module doesn't do) which is better left to the
>> parser.
>
> Maybe there are some lessons here that the TextIO module could learn?
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Ron Adam 



On 06/17/2013 05:18 PM, Greg Ewing wrote:

I'm still not convinced that this is necessary or desirable
behaviour. I can understand the parser doing this as a
workaround before we had universal newlines, but now that
we do, I'd expect any Python string to already have newlines
converted to their canonical representation, and that any CRs
it contains are meant to be there. The parser shouldn't need
to do newline translation a second time.



It's the other way around.

Eval and exec should generate the same results as pythons compiler with the 
same input, including errors and exceptions.  The only way we can have that 
is if eval and exec parses everything the same way.


It's the first parsing that needs to be avoided or compensated for in these 
cases.  Raw strings (my preference) works for string literals, or you can 
escape the escape codes so they are still individual characters after the 
first translation.  Or read the code directly from a file rather than 
importing it.


For example, if you wrote your own python console program, you would want 
all the errors and exceptions to come from eval, including those for bad 
strings.  You would still need to feed the bad strings to eval.  If you 
don't then you won't get the same output from eval as the compiler does.


Cheers,
   Ron

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Benjamin Peterson 
2013/6/17 Guido van Rossum :
> On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson  
> wrote:
>> 2013/6/17 Guido van Rossum :
>>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson  
>>> wrote:
>>> What exactly does the parser handles better than the io module? Is it
>>> just the coding cookies? I suppose that works as long as the file is
>>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
>>> would fail pretty badly if it was UTF-16 (and yes, that's an
>>> abominable encoding for other reasons :-).
>>
>> The coding cookie is the main one. In fact, if you can't parse that,
>> you don't really know what encoding to open the file with at all.
>> There's also small things like BOM handling (you have to use the
>> utf-16-sig encoding with TextIO to get it removed) and defaulting to
>> UTF-8 (which the io module doesn't do) which is better left to the
>> parser.
>
> Maybe there are some lessons here that the TextIO module could learn?

UTF-8 by default would be great, but that ship has sailed.

Reading Python coding cookies is outside the purview of TextIOWrapper.
However, it would be good to have a function in the stdlib to read a
python source file to Unicode; I've definitely implemented that
several times.



--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-17 Thread Victor Stinner 
2013/6/16 Antoine Pitrou :
> On Sun, 16 Jun 2013 01:48:06 +0200
> Victor Stinner  wrote:
>> I just create the issue http://bugs.python.org/issue18227: "Use Python
>> memory allocators in external libraries like zlib or OpenSSL".
>>
>> Is it possible to detect if Python is used as a standalone application
>> (the classic "python" program) or if Python is embedded? If it is
>> possible, we can modify the "global" memory allocators of a library.
>
> The question is why you want to do so, not how/whether to do it.

I want to be able to track the memory usage of all Python memory, even
in external libraries, and/or use a custom memory allocator, even in
external libraries.

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Éric Araujo 
Le 17/06/2013 20:49, Benjamin Peterson a écrit :
> Reading Python coding cookies is outside the purview of TextIOWrapper.
> However, it would be good to have a function in the stdlib to read a
> python source file to Unicode; I've definitely implemented that
> several times.

IIUC you want http://docs.python.org/3/library/tokenize#tokenize.open
(3.2+).

Regards
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Benjamin Peterson 
2013/6/17 Éric Araujo :
> Le 17/06/2013 20:49, Benjamin Peterson a écrit :
>> Reading Python coding cookies is outside the purview of TextIOWrapper.
>> However, it would be good to have a function in the stdlib to read a
>> python source file to Unicode; I've definitely implemented that
>> several times.
>
> IIUC you want http://docs.python.org/3/library/tokenize#tokenize.open
> (3.2+).

Yep. :)



--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod

2013-06-17 Thread Eli Bendersky 
On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka wrote:

> 14.06.13 11:46, Antoine Pitrou написав(ла):
>
>> On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
>> raymond.hettinger  wrote:
>>
>>> http://hg.python.org/cpython/**rev/5accb0ac8bfb
>>> changeset:   84116:5accb0ac8bfb
>>>Fix comment blocks.  Adjust blocksize to a power-of-two for better
>>> divmod computations.
>>>
>>
>> Is there any rationale for changing the heuristic from "fits in a whole
>> number of cachelines" to "allows fast division by the blocksize"?
>>
>> I personally would prefer if such changes were justified a bit more
>> than by a one-liner changeset message without any corresponding open
>> issue.
>>
>
> I share the doubts of Antoine and I was going to write the same comment. I
> thought there were good reasons for previous code. What has changed?
>

I agree it would be interesting to hear about the reasons for the change.
Raymond?

Eli
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod

2013-06-17 Thread Ethan Furman 

On 06/17/2013 09:05 PM, Eli Bendersky wrote:




On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka mailto:[email protected]>> wrote:

14.06.13 11:46, Antoine Pitrou написав(ла):

On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
raymond.hettinger mailto:[email protected]>> wrote:

http://hg.python.org/cpython/__rev/5accb0ac8bfb 

changeset:   84116:5accb0ac8bfb
Fix comment blocks.  Adjust blocksize to a power-of-two for 
better divmod computations.


Is there any rationale for changing the heuristic from "fits in a whole
number of cachelines" to "allows fast division by the blocksize"?

I personally would prefer if such changes were justified a bit more
than by a one-liner changeset message without any corresponding open
issue.


I share the doubts of Antoine and I was going to write the same comment. I 
thought there were good reasons for
previous code. What has changed?


I agree it would be interesting to hear about the reasons for the change. 
Raymond?


Asking as a learner:  are such non-bugfix changes appropriate for the 2.7 line?

--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com