Re: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default): Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from

2014-05-12 Thread R. David Murray 
These changes appear to have caused several builbot failures, and
there doesn't appear to be a bugs.python.org issue to report it to.
One failure example:

http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119

test_asyncio fails similarly for me on tip.

On Mon, 12 May 2014 19:05:19 +0200, guido.van.rossum 
 wrote:
> http://hg.python.org/cpython/rev/2af5a52b9b87
> changeset:   90663:2af5a52b9b87
> parent:  90661:9493fdad2a75
> parent:  90662:909ea8cc86bb
> user:Guido van Rossum 
> date:Mon May 12 10:05:04 2014 -0700
> summary:
>   Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) 
> from pipe may hang if data exceeds buffer limit.
> 
> files:
>   Lib/asyncio/streams.py|  17 --
>   Lib/test/test_asyncio/test_streams.py |  36 +++
>   2 files changed, 47 insertions(+), 6 deletions(-)
> 
> 
> diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py
> --- a/Lib/asyncio/streams.py
> +++ b/Lib/asyncio/streams.py
> @@ -419,12 +419,17 @@
>  return b''
>  
>  if n < 0:
> -while not self._eof:
> -self._waiter = self._create_waiter('read')
> -try:
> -yield from self._waiter
> -finally:
> -self._waiter = None
> +# This used to just loop creating a new waiter hoping to
> +# collect everything in self._buffer, but that would
> +# deadlock if the subprocess sends more than self.limit
> +# bytes.  So just call self.read(self._limit) until EOF.
> +blocks = []
> +while True:
> +block = yield from self.read(self._limit)
> +if not block:
> +break
> +blocks.append(block)
> +return b''.join(blocks)
>  else:
>  if not self._buffer and not self._eof:
>  self._waiter = self._create_waiter('read')
> diff --git a/Lib/test/test_asyncio/test_streams.py 
> b/Lib/test/test_asyncio/test_streams.py
> --- a/Lib/test/test_asyncio/test_streams.py
> +++ b/Lib/test/test_asyncio/test_streams.py
> @@ -1,7 +1,9 @@
>  """Tests for streams.py."""
>  
>  import gc
> +import os
>  import socket
> +import sys
>  import unittest
>  from unittest import mock
>  try:
> @@ -583,6 +585,40 @@
>  server.stop()
>  self.assertEqual(msg, b"hello world!\n")
>  
> [email protected](sys.platform == 'win32', "Don't have pipes")
> +def test_read_all_from_pipe_reader(self):
> +# See Tulip issue 168.  This test is derived from the example
> +# subprocess_attach_read_pipe.py, but we configure the
> +# StreamReader's limit so that twice it is less than the size
> +# of the data writter.  Also we must explicitly attach a child
> +# watcher to the event loop.
> +
> +watcher = asyncio.get_child_watcher()
> +watcher.attach_loop(self.loop)
> +
> +code = """\
> +import os, sys
> +fd = int(sys.argv[1])
> +os.write(fd, b'data')
> +os.close(fd)
> +"""
> +rfd, wfd = os.pipe()
> +args = [sys.executable, '-c', code, str(wfd)]
> +
> +pipe = open(rfd, 'rb', 0)
> +reader = asyncio.StreamReader(loop=self.loop, limit=1)
> +protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop)
> +transport, _ = self.loop.run_until_complete(
> +self.loop.connect_read_pipe(lambda: protocol, pipe))
> +
> +proc = self.loop.run_until_complete(
> +asyncio.create_subprocess_exec(*args, pass_fds={wfd}, 
> loop=self.loop))
> +self.loop.run_until_complete(proc.wait())
> +
> +os.close(wfd)
> +data = self.loop.run_until_complete(reader.read(-1))
> +self.assertEqual(data, b'data')
> +
>  
>  if __name__ == '__main__':
>  unittest.main()
> 
> -- 
> Repository URL: http://hg.python.org/cpython
> ___
> Python-checkins mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-checkins
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert.

2014-05-12 Thread Eric Snow 
On Sun, May 11, 2014 at 7:57 AM, Steve Dower  wrote:
> For those who missed the earlier discussions, Martin v. Löwis has handed
> over responsibility for the Windows installers. It sounds like Brett Cannon
> and I are both in a position to build 2.7 right now, and I hope to simplify
> the setup required for 3.5 so that anyone can produce and test the
> installers. (Martin is going to look after the 3.4.x builds.)

You're a welcome addition to the group!

>
> Obviously I'm also here to help out with Windows in general. I haven't quite
> managed to get 50% time from my employer (maybe 10%), but my management at
> least is very supportive of my participation and keen to keep Python running
> well.

That's great news.  Hopefully it's a growing trend. :)

-eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Questions regarding Windows buildbots

2014-05-12 Thread Claudiu Popa 
Hello!

I'm working on a patch for issue bugs.python.org/issue8579 (Add
missing tests for FlushKey, LoadKey, and SaveKey in winreg). This
issue requires the SeBackupPrivilege in order to use LoadKey and
SaveKey. While acquiring the privilege
isn't very complicated using ctypes, it fails with
ERROR_NOT_ALL_ASSIGNED (1300) when the
user has Administrative privileges, but it's not an Administrator,
problem which can be eluded by
running the script with elevated privileges. This leads me to a couple
of questions:

- should a Windows test be skipped if it can't acquire a certain privilege?

- If we can acquire the privilege by elevating our process, does the
Windows buildbots have UAC
 enabled and if so, how's the notification setting configured? For
instance, elevating a process will
 trigger a new UAC window with the message "Do you want to allow the
following program from an unknown publisher to make  changes to this
computer?" on the recommended configuration, but this doesn't happen
when the configuration is set to  "Never notify".


Thank you.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Questions regarding Windows buildbots

2014-05-12 Thread Brian Curtin 
On Mon, May 12, 2014 at 5:16 PM, Claudiu Popa  wrote:
> Hello!
>
> I'm working on a patch for issue bugs.python.org/issue8579 (Add
> missing tests for FlushKey, LoadKey, and SaveKey in winreg). This
> issue requires the SeBackupPrivilege in order to use LoadKey and
> SaveKey. While acquiring the privilege
> isn't very complicated using ctypes, it fails with
> ERROR_NOT_ALL_ASSIGNED (1300) when the
> user has Administrative privileges, but it's not an Administrator,
> problem which can be eluded by
> running the script with elevated privileges. This leads me to a couple
> of questions:
>
> - should a Windows test be skipped if it can't acquire a certain privilege?

Yes. Check out any of the os.symlink tests - they're currently skipped
when the symlink privilege isn't held.

> - If we can acquire the privilege by elevating our process, does the
> Windows buildbots have UAC
>  enabled and if so, how's the notification setting configured? For
> instance, elevating a process will
>  trigger a new UAC window with the message "Do you want to allow the
> following program from an unknown publisher to make  changes to this
> computer?" on the recommended configuration, but this doesn't happen
> when the configuration is set to  "Never notify".

That probably depends on how each machine is setup. If they happen to
get blocked on any individual slave, we'll just have to ask the owner
to change that setting.

Currently there are no Windows build slaves running as administrator.
I used to have one but the machine died and I never replaced it. I
also said a few months ago that I would get one setup again, but that
hasn't happened yet. I can get a new machine up and running but
probably not until next week as I'm at a conference.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python process creation overhead

2014-05-12 Thread Gregory Szorc 
On 5/10/2014 2:46 PM, Victor Stinner wrote:
> Le 10 mai 2014 22:51, "Gregory Szorc"  > a écrit :
>> Furthermore, Python 3 appears to be >50% slower than Python 2.
> 
> Please mention the minor version. It looks like you compared 2.7
> and 3.3. Please test 3.4, we made interesting progress on the
> startup time.
> 
> There is still something to do, especially on OS X. Depending on
> the OS, different modules are loaded and some functions are
> implemented differently.

3.4.0 does appear to be faster than 3.3.5 on Linux - `python -c ''` is
taking ~50ms (as opposed to ~60ms) on my i7-2600K. Great to see!

But 3.4.0 is still slower than 2.7.6. And all versions of CPython are
over 3x slower than Perl 5.18.2. This difference amounts to minutes of
CPU time when thousands of processes are involved. That seems
excessive to me.

Why can't Python start as quickly as Perl or Ruby?
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default): Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from

2014-05-12 Thread Eric Snow 
On Mon, May 12, 2014 at 12:43 PM, R. David Murray  wrote:
> These changes appear to have caused several builbot failures, and
> there doesn't appear to be a bugs.python.org issue to report it to.
> One failure example:
>
> 
> http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119
>
> test_asyncio fails similarly for me on tip.

Same for me on 3.4.

-eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python process creation overhead

2014-05-12 Thread dw+python-dev 
On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote:

> Why can't Python start as quickly as Perl or Ruby?

On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from
81ms startup to 20ms by specifying -S, which disables site.py.

Oblitering the .pth files immediately knocks 10ms off regular startup. I
guess the question isn't why Python is slower than perl, but what
aspects of site.py could be cached, reimplemented, or stripped out
entirely.  I'd personally love to see .pth support removed.


David
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default): Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from

2014-05-12 Thread Guido van Rossum 
Sorry about that. I will look into it soon, but won't have time right away.
(Also I missed David's earlier message.)
On May 12, 2014 5:14 PM, "Eric Snow"  wrote:

> On Mon, May 12, 2014 at 12:43 PM, R. David Murray 
> wrote:
> > These changes appear to have caused several builbot failures, and
> > there doesn't appear to be a bugs.python.org issue to report it to.
> > One failure example:
> >
> >
> http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119
> >
> > test_asyncio fails similarly for me on tip.
>
> Same for me on 3.4.
>
> -eric
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python process creation overhead

2014-05-12 Thread Nick Coghlan 
On 13 May 2014 10:19,   wrote:
> On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote:
>
>> Why can't Python start as quickly as Perl or Ruby?
>
> On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from
> 81ms startup to 20ms by specifying -S, which disables site.py.
>
> Oblitering the .pth files immediately knocks 10ms off regular startup. I
> guess the question isn't why Python is slower than perl, but what
> aspects of site.py could be cached, reimplemented, or stripped out
> entirely.  I'd personally love to see .pth support removed.

The startup code is currently underspecified and underdocumented, and
quite fragile as a result. It represents 20+ years of organic growth
without any systematic refactoring to simplify and streamline things.

That's what PEP 432 aims to address, and is something I now expect to
have time to get back to for Python 3.5. And yes, one thing those
changes should enable is the creation of system Python runtimes on
Linux that initialise faster than the current implementation.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com