from:"M.\-A. Lemburg"

On 04.04.2015 02:49, Donald Stufft wrote:
 
 On Apr 3, 2015, at 6:38 PM, M.-A. Lemburg m...@egenix.com wrote:

 On 04.04.2015 00:14, Steve Dower wrote:
 The thing is, that's exactly the same goodness as Authenticode gives, 
 except everyone gets that for free and meanwhile you're the only one who 
 has admitted to using GPG on Windows :)

 Basically, what I want to hear is that GPG sigs provide significantly 
 better protection than hashes (and I can provide better than MD5 for all 
 files if it's useful), taking into consideration that (I assume) I'd have 
 to obtain a signing key for GPG and unless there's a CA involved like there 
 is for Authenticode, there's no existing trust in that key.

 Hashes only provide checks against file corruption (and then
 only if you can trust the hash values). GPG provides all the
 benefits of public key encryption on arbitrary files (not just
 code).

 The main benefit in case of downloadable installers is to
 be able to make sure that the files are authentic, meaning that
 they were created and signed by the people listed as packagers.

 There is no CA infrastructure involved as for SSL certificates
 or Authenticode, but it's easy to get the keys from key servers
 given the key signatures available from python.org's download
 pages.
 
 FTR if we’re relying on people to get the GPG keys from the download
 pages then there’s no additional benefit over just using a hash
 published on the same page.

Well, it's still better than just the hashes...

 In order to get additional benefit we’d need to get Steve’s key
 signed by enough people to get him into the strong set.

...but having the key signed by fellow core devs will certainly
add more goodness :-)

 If you want to sign a package file using GPG, you will need
 to create your own key, upload it to the key servers and then
 place the signature up on the download page.

 Relying only on Authenticode for Windows installers would
 result in a break in technology w/r to the downloads we
 make available for Python, since all other files are (usually)
 GPG signed:

 https://www.python.org/ftp/python/3.4.3/

 Cheers,
 --
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 

 ::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


 Cheers,
 Steve

 Top-posted from my Windows Phone
 
 From: M.-A. Lemburgmailto:m...@egenix.com
 Sent: ‎4/‎3/‎2015 10:55
 To: Steve Dowermailto:steve.do...@microsoft.com; Larry 
 Hastingsmailto:la...@hastings.org; Python 
 Devmailto:python-dev@python.org; 
 python-committersmailto:python-committ...@python.org
 Subject: Re: [python-committers] [Python-Dev] Do we need to sign Windows 
 files with GnuPG?

 On 03.04.2015 19:35, Steve Dower wrote:
 My Windows development days are firmly behind me. So I don't really have 
 an
 opinion here. So I put it to you, Windows Python developers: do you care 
 about
 GnuPG signatures on Windows-specific files? Or do you not care?

 The later replies seem to suggest that they are general goodness that 
 nobody on Windows will use. If someone convinces me (or steamrolls me, 
 that's fine too) that the goodness of GPG is better than a hash then I'll 
 look into adding it into the process. Otherwise I'll happily add hash 
 generation into the upload process (which I'm going to do anyway for the 
 ones displayed on the download page).

 FWIW: I regularly check the GPG sigs on all important downloaded
 files, regardless of which platform they target, including the
 Windows installers for Python or any other Windows installers
 I use which provide such sigs.

 The reason is simple:
 The signature is a proof of authenticity which is not bound to
 a particular file format or platform and before running .exes
 it's good to know that they were built by the right people and
 not manipulated by trojans, viruses or malicious proxies.

 Is that a good enough reason to continue providing the GPG
 sigs or do you need more proof of goodness ? ;-)

 --
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 

 ::: Try our new mxODBC.Connect Python Database

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

On 04.04.2015 16:41, Steve Dower wrote:
 Relying only on Authenticode for Windows installers would result in a break 
 in technology w/r to the downloads we make available for Python, since all 
 other files are (usually) GPG signed
 
 This is the point of this discussion. I'm willing to make such a break 
 because I believe Authenticode is so much more convenient for end users that 
 it isn't worth producing GPG signatures. So far, the responses seem to be:
 
 I'd use them on Windows x1
 I'd consider using them on another OS x2-3
 Please don't change everyone else
 
 At least that's the impression I'm getting, so I hope that helps clarify why 
 I'm still not convinced it's that critical.

Just to clarify:

I have absolutely nothing against using Authenticode on Windows :-)

I'm only trying to convince you that *additionally* providing GPG
sigs for Windows downloads is a good thing and we should not stop
doing this, since it makes verification of downloaded files
easier. It's not hard to do, can be automated and provides additional
security which can be verified on any platform, not only Windows.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

On 04.04.2015 21:02, Kurt B. Kaiser wrote:
 For the record, that is a Symantec/Verisign code signing certificate. We
 paid $1123 for it last April.  It expires April 2017.
 
 If you don't switch to a different vendor, e.g. startssl, please contact
 me for renewal in 2017.

FWIW: The PSF mostly uses StartSSL nowadays and they also support
code signing certificates. Given that this option is a lot cheaper
than Verisign, I think we should switch, unless there are
significant reasons not to. We should revisit this in 2017.

 KBK
 
 On Sat, Apr 4, 2015, at 10:35 AM, Steve Dower wrote:
 Small clarification: there certificates *are* the same format as for SSL,
 and OpenSSL it's able to validate them in the same way as well as
 generate them (but not extract embedded ones, AFAICT). But generally SSL
 certificates are not marked as suitable for code signing so you need to
 buy a separate one.

 Both Martin and I have the PSF's code signing cert private key, which is
 how we can sign with the Python Software Foundation name. The public
 key is embedded into every signed file, just like an SSL cert is attached
 to a site or an S/MIME cert is embedded in a signed email.

 Cheers,
 Steve

 Top-posted from my Windows Phone
 
 From: Steve Dowermailto:steve.do...@microsoft.com
 Sent: ‎4/‎4/‎2015 7:25
 To: Wes Turnermailto:wes.tur...@gmail.com; M. -A.
 Lemburgmailto:m...@egenix.com
 Cc: python-committersmailto:python-committ...@python.org;
 Python-Devmailto:python-dev@python.org
 Subject: Re: [python-committers] [Python-Dev] Do we need to sign Windows
 files with GnuPG?

 Authenticode does not have a PKI

 If you got that from this discussion, I need everyone to at least skim
 read this: 
 https://msdn.microsoft.com/en-us/library/ie/ms537361(v=vs.85).aspx

 Authenticode uses the same certificate infrastructure as SSL (note: not
 the same certificates). As I see it, anyone running on Windows has access
 to verification that is at least as good as GPG, and the only people who
 would benefit from GPG sigs are those checking Windows files on another
 OS or those with an existing GPG workflow on Windows (before this thread,
 I knew nobody who used GPG on Windows for anything, so forgive me for
 thinking this is very rare).

 Cheers,
 Steve

 Top-posted from my Windows Phone
 
 From: Wes Turnermailto:wes.tur...@gmail.com
 Sent: ‎4/‎4/‎2015 6:42
 To: M. -A. Lemburgmailto:m...@egenix.com
 Cc: Python-Devmailto:python-dev@python.org;
 python-committersmailto:python-committ...@python.org; Larry
 Hastingsmailto:la...@hastings.org; Steve
 Dowermailto:steve.do...@microsoft.com
 Subject: Re: [Python-Dev] [python-committers] Do we need to sign Windows
 files with GnuPG?


 So, AFAIU from this discussion:

 * Authenticode does not have a PKI
 * GPG does have PKI
 * ASC signatures are signed checksums

 As far as downstream packaging on Windows (people who should/could be
 subscribed to release ANNs):

 For Choclatey NuGet:

 * https://chocolatey.org/packages/python
 * https://chocolatey.org/packages/python.x86
 * https://chocolatey.org/packages/python2
 * https://chocolatey.org/packages/python-x86_32
 * https://chocolatey.org/packages/python3

 Python(x,y):

 * https://code.google.com/p/pythonxy/

 For Anaconda (the MS Azure chosen python distribution):

 * http://docs.continuum.io/anaconda/install.html#windows-install

 ...

 These should/could/are checking GPG signatures for Windows packages
 downstream.

 http://www.scipy.org/install.html

 On Apr 3, 2015 5:38 PM, M.-A. Lemburg
 m...@egenix.commailto:m...@egenix.com wrote:
 On 04.04.2015 00:14, Steve Dower wrote:
 The thing is, that's exactly the same goodness as Authenticode gives, 
 except everyone gets that for free and meanwhile you're the only one who 
 has admitted to using GPG on Windows :)

 Basically, what I want to hear is that GPG sigs provide significantly 
 better protection than hashes (and I can provide better than MD5 for all 
 files if it's useful), taking into consideration that (I assume) I'd have 
 to obtain a signing key for GPG and unless there's a CA involved like there 
 is for Authenticode, there's no existing trust in that key.

 Hashes only provide checks against file corruption (and then
 only if you can trust the hash values). GPG provides all the
 benefits of public key encryption on arbitrary files (not just
 code).

 The main benefit in case of downloadable installers is to
 be able to make sure that the files are authentic, meaning that
 they were created and signed by the people listed as packagers.

 There is no CA infrastructure involved as for SSL certificates
 or Authenticode, but it's easy to get the keys from key servers
 given the key signatures available from python.orghttp://python.org's
 download
 pages.

 If you want to sign a package file using GPG, you will need
 to create your own key, upload it to the key servers and then
 place the signature up on the download page

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

On 04.04.2015 21:49, Kurt B. Kaiser wrote:
 
 
 On Sat, Apr 4, 2015, at 03:35 PM, M.-A. Lemburg wrote:
 On 04.04.2015 21:02, Kurt B. Kaiser wrote:
 For the record, that is a Symantec/Verisign code signing
 certificate. We paid $1123 for it last April.  It expires
 April 2017.

 If you don't switch to a different vendor, e.g. startssl, please
 contact me for renewal in 2017.

 FWIW: The PSF mostly uses StartSSL nowadays and they also support code
 signing certificates. Given that this option is a lot cheaper than
 Verisign, I think we should switch, unless there are significant
 reasons not to. We should revisit this in 2017.
 
 Agree - apparently the starlssl process for getting a signing cert is
 complex/obscure, so we should start early.

Not really. Once you have the org verification it's really easy.

 Let me know if I can help providing PSF organization verification.

I already completed that for the current cycle.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

2015-04-03 Thread M.-A. Lemburg

On 03.04.2015 11:56, Larry Hastings wrote:
 My Windows development days are firmly behind me.  So I don't really have an 
 opinion here.  So I put
 it to you, Windows Python developers: do you care about GnuPG signatures on 
 Windows-specific files? 
 Or do you not care?

Regardless of target platform, I firmly believe we should (continue to)
GPG sign all distribution files as well as provide hash files/values
for them.

This is very useful to detect corrupted downloads or files which
were not created by the original packagers.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

2015-04-03 Thread M.-A. Lemburg

On 03.04.2015 19:35, Steve Dower wrote:
 My Windows development days are firmly behind me. So I don't really have an
 opinion here. So I put it to you, Windows Python developers: do you care 
 about
 GnuPG signatures on Windows-specific files? Or do you not care?
 
 The later replies seem to suggest that they are general goodness that nobody 
 on Windows will use. If someone convinces me (or steamrolls me, that's fine 
 too) that the goodness of GPG is better than a hash then I'll look into 
 adding it into the process. Otherwise I'll happily add hash generation into 
 the upload process (which I'm going to do anyway for the ones displayed on 
 the download page).

FWIW: I regularly check the GPG sigs on all important downloaded
files, regardless of which platform they target, including the
Windows installers for Python or any other Windows installers
I use which provide such sigs.

The reason is simple:
The signature is a proof of authenticity which is not bound to
a particular file format or platform and before running .exes
it's good to know that they were built by the right people and
not manipulated by trojans, viruses or malicious proxies.

Is that a good enough reason to continue providing the GPG
sigs or do you need more proof of goodness ? ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

2015-04-03 Thread M.-A. Lemburg

On 04.04.2015 00:14, Steve Dower wrote:
 The thing is, that's exactly the same goodness as Authenticode gives, except 
 everyone gets that for free and meanwhile you're the only one who has 
 admitted to using GPG on Windows :)
 
 Basically, what I want to hear is that GPG sigs provide significantly better 
 protection than hashes (and I can provide better than MD5 for all files if 
 it's useful), taking into consideration that (I assume) I'd have to obtain a 
 signing key for GPG and unless there's a CA involved like there is for 
 Authenticode, there's no existing trust in that key.

Hashes only provide checks against file corruption (and then
only if you can trust the hash values). GPG provides all the
benefits of public key encryption on arbitrary files (not just
code).

The main benefit in case of downloadable installers is to
be able to make sure that the files are authentic, meaning that
they were created and signed by the people listed as packagers.

There is no CA infrastructure involved as for SSL certificates
or Authenticode, but it's easy to get the keys from key servers
given the key signatures available from python.org's download
pages.

If you want to sign a package file using GPG, you will need
to create your own key, upload it to the key servers and then
place the signature up on the download page.

Relying only on Authenticode for Windows installers would
result in a break in technology w/r to the downloads we
make available for Python, since all other files are (usually)
GPG signed:

https://www.python.org/ftp/python/3.4.3/

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


 Cheers,
 Steve
 
 Top-posted from my Windows Phone
 
 From: M.-A. Lemburgmailto:m...@egenix.com
 Sent: ‎4/‎3/‎2015 10:55
 To: Steve Dowermailto:steve.do...@microsoft.com; Larry 
 Hastingsmailto:la...@hastings.org; Python 
 Devmailto:python-dev@python.org; 
 python-committersmailto:python-committ...@python.org
 Subject: Re: [python-committers] [Python-Dev] Do we need to sign Windows 
 files with GnuPG?
 
 On 03.04.2015 19:35, Steve Dower wrote:
 My Windows development days are firmly behind me. So I don't really have an
 opinion here. So I put it to you, Windows Python developers: do you care 
 about
 GnuPG signatures on Windows-specific files? Or do you not care?

 The later replies seem to suggest that they are general goodness that nobody 
 on Windows will use. If someone convinces me (or steamrolls me, that's fine 
 too) that the goodness of GPG is better than a hash then I'll look into 
 adding it into the process. Otherwise I'll happily add hash generation into 
 the upload process (which I'm going to do anyway for the ones displayed on 
 the download page).
 
 FWIW: I regularly check the GPG sigs on all important downloaded
 files, regardless of which platform they target, including the
 Windows installers for Python or any other Windows installers
 I use which provide such sigs.
 
 The reason is simple:
 The signature is a proof of authenticity which is not bound to
 a particular file format or platform and before running .exes
 it's good to know that they were built by the right people and
 not manipulated by trojans, viruses or malicious proxies.
 
 Is that a good enough reason to continue providing the GPG
 sigs or do you need more proof of goodness ? ;-)
 
 --
 Marc-Andre Lemburg
 eGenix.com
 
 Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 
 
 ::: Try our new mxODBC.Connect Python Database Interface for free ! 
 
 
eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Encoding of PyFrameObject members

2015-02-06 Thread M.-A. Lemburg

On 06.02.2015 00:27, Francis Giraldeau wrote:
 I need to access frame members from within a signal handler for tracing
 purpose. My first attempt to access co_filename was like this (omitting
  error checking):
 
 PyFrameObject *frame = PyEval_GetFrame();
 PyObject *ob = PyUnicode_AsUTF8String(frame-f_code-co_filename)
 char *str = PyBytes_AsString(ob)
 
 However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(),
 which is not reentrant. If the signal handler nest over PyObject_Malloc(),
 it causes a segfault, and it could also deadlock.
 
 Instead, I access members directly:
 char *str = PyUnicode_DATA(frame-f_code-co_filename);
 size_t len = PyUnicode_GET_DATA_SIZE(frame-f_code-co_filename);
 
 Is it safe to assume that unicode objects co_filename and co_name are
 always UTF-8 data for loaded code? I looked at the PyTokenizer_FromString()
 and it seems to convert everything to UTF-8 upfront, and I would like to
 make sure this assumption is valid.

The macros won't work in all cases, as they don't pay attention
to the different kinds used in the Unicode implementation.

I don't think there's any API you can use to extract the
underlying data without going through PyObject_Malloc()
at some point (you may be lucky if there already is a
UTF-8 version available, but it's not guaranteed).

I guess your best bet is to write your own UTF-8
codec which then copies the data to a buffer that
you can control. Have a look at Objects/stringlib/codecs.h:
utf8_encode.

Alternatively, you can copy the data to a Py_UCS4 buffer
which you allocate using code such as this (untested,
adapted from the UTF-8 encoder):

Py_UCS4 *p;
enum PyUnicode_Kind repkind;
void *repdata;
Py_ssize_t repsize, k;

if (PyUnicode_READY(rep)  0)
goto error;
repkind = PyUnicode_KIND(rep);
repdata = PyUnicode_DATA(rep);
repsize = PyUnicode_GET_LENGTH(rep);

p = malloc((repsize + 1) * sizeof(Py_UCS4));
for(k=0; krepsize; k++) {
*p++ = PyUnicode_READ(repkind, repdata, k);
}
/* 0-terminate */
*p++ = 0;

...

free(p);

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 06 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python Be Built Without Distutils

2015-01-24 Thread M.-A. Lemburg

On 24.01.2015 21:23, Zachary Ware wrote:
 On Saturday, January 24, 2015, Brett Cannon br...@python.org wrote:
 
 On Fri Jan 23 2015 at 5:45:28 PM Gregory P. Smith g...@krypto.org
 javascript:_e(%7B%7D,'cvml','g...@krypto.org'); wrote:

 On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg m...@egenix.com
 javascript:_e(%7B%7D,'cvml','m...@egenix.com'); wrote:

 On 23.01.2015 19:48, Matthias Klose wrote:
 On 01/23/2015 06:30 PM, Cyd Haselton wrote:
 Related to my earlier question regarding building Python on Android
 and an undefined reference to dlopen error...I have the following
 question:  Is it possible to build and install Python without having
 to build and install...or use...distutils?

 Some background:
 I can build the python interpreter on my device, and I can build a
 bunch of modules.  The problem appears when make reaches the part
 where setup.py is used to build and import modules...specifically when
 setup.py attempts to import distutils.core.

 you can do this using Setup.local. This works for me building
 additional
 extensions as builtins.  It might require some tweaking to build
 everything.

 You may want to have a look at the Setup files we're using
 in eGenix PyRun, which uses them to force static builds of the
 various built-in extensions.

 Look for these files:

 PyRun/Runtime/Setup.PyRun-2.7
 PyRun/Runtime/Setup.PyRun-3.4

 in the source archives:

 http://www.egenix.com/products/python/PyRun/

 Otoh, I would like to get rid off the setup.py altogether (/me ducks
 ...).

 Why ? It's great for finding stuff on your system and configuring
 everything without user intervention (well, most of the time :-)).


 Because our setup.py is a nightmare of arbitrary code run in a linear
 fashion with ad-hoc checks for things that are unlike how any other project
 on the planet determines what is available on your system.  It may have
 seemed great when it was created in 2001.  It really shows its age now.

 It defeats build parallelism and dependency declaration.
 It also prevents cross compilation.

 Building an interpreter with a limited standard library on your build
 host so that you can run said interpreter to have it drive the remainder of
 your build is way more self hosting that we rightfully need to be for
 CPython.


 So are you suggesting to add the build rules to configure and the Makefile
 -- and Windows build file -- in order to drop setup.py?

 
 Windows already doesn't use setup.py. There are a lot more modules built-in
 on Windows, and others have their own project files; distutils isn't used
 at all.

The Windows installers comes with all dependencies included.

On other systems, this is not the case and so determining the
various settings is done using Python, which is a lot easier to
do than e.g. relying on autoconf finding everything and setting up
the correct compile options for each built-in.

All that said: if you can come up with a better system that's
both easy to maintain and provides the same level of user
friendliness, I'm sure this would be considered.

BTW: Parallel execution, cross compilation can be added
to setup.py. I don't think parallel execution is all
that important, but cross compilation would certainly be an
interesting feature to have (this currently has to be done
using the Setup.in approach) and an option to build the
modules statically linked to the interpreter would also be
nice.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 24 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python Be Built Without Distutils

2015-01-23 Thread M.-A. Lemburg

On 23.01.2015 19:48, Matthias Klose wrote:
 On 01/23/2015 06:30 PM, Cyd Haselton wrote:
 Related to my earlier question regarding building Python on Android
 and an undefined reference to dlopen error...I have the following
 question:  Is it possible to build and install Python without having
 to build and install...or use...distutils?

 Some background:
 I can build the python interpreter on my device, and I can build a
 bunch of modules.  The problem appears when make reaches the part
 where setup.py is used to build and import modules...specifically when
 setup.py attempts to import distutils.core.
 
 you can do this using Setup.local. This works for me building additional
 extensions as builtins.  It might require some tweaking to build everything.

You may want to have a look at the Setup files we're using
in eGenix PyRun, which uses them to force static builds of the
various built-in extensions.

Look for these files:

PyRun/Runtime/Setup.PyRun-2.7
PyRun/Runtime/Setup.PyRun-3.4

in the source archives:

http://www.egenix.com/products/python/PyRun/

 Otoh, I would like to get rid off the setup.py altogether (/me ducks ...).

Why ? It's great for finding stuff on your system and configuring
everything without user intervention (well, most of the time :-)).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 23 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python Be Built Without Distutils

2015-01-23 Thread M.-A. Lemburg

On 23.01.2015 21:56, Cyd Haselton wrote:
 On Fri, Jan 23, 2015 at 1:19 PM, M.-A. Lemburg m...@egenix.com wrote:
 On 23.01.2015 19:48, Matthias Klose wrote:
 On 01/23/2015 06:30 PM, Cyd Haselton wrote:
 Related to my earlier question regarding building Python on Android
 and an undefined reference to dlopen error...I have the following
 question:  Is it possible to build and install Python without having
 to build and install...or use...distutils?

 Some background:
 I can build the python interpreter on my device, and I can build a
 bunch of modules.  The problem appears when make reaches the part
 where setup.py is used to build and import modules...specifically when
 setup.py attempts to import distutils.core.

 you can do this using Setup.local. This works for me building additional
 extensions as builtins.  It might require some tweaking to build everything.

 You may want to have a look at the Setup files we're using
 in eGenix PyRun, which uses them to force static builds of the
 various built-in extensions.

 Look for these files:

 PyRun/Runtime/Setup.PyRun-2.7
 PyRun/Runtime/Setup.PyRun-3.4

 in the source archives:

 http://www.egenix.com/products/python/PyRun/
 
 Do you have a browseable git repo?

No, but we're thinking of mirroring our open-source packages on
Github or Bitbucket.

 Otoh, I would like to get rid off the setup.py altogether (/me ducks ...).

 Why ? It's great for finding stuff on your system and configuring
 everything without user intervention (well, most of the time :-)).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 23 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile Python on Windows (OpenSSL)

2015-01-14 Thread M.-A. Lemburg

On 13.01.2015 23:50, Victor Stinner wrote:
 2015-01-13 23:46 GMT+01:00 M.-A. Lemburg m...@egenix.com:
 Just a note of caution: for older preview releases of VS the
 only way to get back to a clean system was to reinstall
 Windows.
 
 Does it mean that it's not possible to have VS 2008 and VS 2015
 installed at the same time?
 
 VS 2008 is required to build Python 2.7.

Steve already answered this basically. It is well possible to
have multiple VS versions installed at the same time. Before
compiling you just need to run the right environment setup batch file
to prepare everything.

I was just referring to *preview* versions of VS. If you use
one of those, chances are that you'll have a hard time upgrading
to the final version of VS. My recommendation is not to use preview
versions on systems that you cannot easily revert to a state
before the preview install.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 14 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-01-09: Released eGenix pyOpenSSL 0.13.7 ...  http://egenix.com/go68
2015-01-20: Python Meeting Duesseldorf ...http://egenix.com/go69

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile Python on Windows (OpenSSL)

2015-01-13 Thread M.-A. Lemburg

On 13.01.2015 23:42, Brian Curtin wrote:
 On Tue, Jan 13, 2015 at 4:36 PM, Victor Stinner
 victor.stin...@gmail.com wrote:
 2015-01-13 23:18 GMT+01:00 Steve Dower steve.do...@microsoft.com:
 Technically, Python 3.5 requires Visual Studio 2015

 For me, it's *very* difficult to find how to install Visual Studio.
 There are many different websites and web pages which mention Visual
 Studio with a lot of versions and flavors (Express, Community,
 Ultimate, etc.).

 Visual Studio 2015 was not released yet :-/

 My VM has only a disk of 40 GB. Only 12 GB are free. I already have VS
 2008 Express and VS 2010 Express installed. I understood that
 Ultimate includes a *lot* of things, not only a C compiler.

 I found a free Visual Studio which is in fact Visual Studio 2013
 Community and I read that it's not free.

 I sent an email to Brian Curtin to ask to renew my MSDN account. He
 didn't reply yet.
 
 I saw that and will send it on, but it's still going to take some time
 to process - usually a week or so.
 
 In the meantime, the first result searching for Visual Studio 2015
 came up with 
 http://www.visualstudio.com/en-us/downloads/visual-studio-2015-downloads-vs.aspx,
 which seems to give you VS2015. I haven't tried to run it since I'm
 not on Windows at the moment, but it looks correct.

Just a note of caution: for older preview releases of VS the
only way to get back to a clean system was to reinstall
Windows.

I don't know whether this will be different with VS 2015,
but if you care for your VM, you should probably create
a snapshot before installing VS 2015 preview to make it
easy to revert back, e.g. to install the final VS 2015
version.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 13 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-01-09: Released eGenix pyOpenSSL 0.13.7 ...  http://egenix.com/go68
2015-01-20: Python Meeting Duesseldorf ...http://egenix.com/go69

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile Python on Windows (OpenSSL)

2015-01-13 Thread M.-A. Lemburg

On 13.01.2015 23:04, Victor Stinner wrote:
 Hi,
 
 To compile Python on Windows, there are a few information in the
 Developer Guide:
 https://docs.python.org/devguide/setup.html#windows-compiling
 
 Python 3.5 now requires Visual Studio 2010 *SP1*, or newer Visual Studio:
 http://bugs.python.org/issue22919#msg233637
 
 I found PCbuild\readme.txt which is not mentionned in the devguide :-/
 https://hg.python.org/cpython/file/56f717235c45/PCbuild/readme.txt
 (at least not on the Windows section of the setup page)
 
 I found some clues to build OpenSSL to be able to build the Python ssl
 module, but I still have issues.
 
 Is there a more complete documentation?
 
 I found how to install svn.exe, perl.exe and nasm.exe, but not how to
 install nmake.exe. I don't know the command to build OpenSSL.
 
 I don't care of building OpenSSL, my goal is only to build the Python
 ssl module. Is there a way to install a development version of OpenSSL
 (.lib files if I remember correctly) from an installer/binary?

If you just need the compiled OpenSSL DLLs and .libs, you
can use the ones we ship with the source package of egenix-pyopenssl:

http://www.egenix.com/products/python/pyOpenSSL/#Download

The files are in openssl-win32 / openssl-win64.

Those dirs also have batch files which we use to build the
OpenSSL libs on Windows.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 13 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-01-09: Released eGenix pyOpenSSL 0.13.7 ...  http://egenix.com/go68
2015-01-20: Python Meeting Duesseldorf ...http://egenix.com/go69

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Redirection of ar.pycon.org

2014-12-18 Thread M.-A. Lemburg

Hi Facunda,

you should either write to webmas...@pycon.org, the
conference ML or me directly, since I'm managing these the
pycon.org subdomains.

 On Thu, Dec 18, 2014, at 10:59, Facundo Batista wrote:
 Hi!

 Don't remember where to ask for changing the redirection of that
 domain name. Somebody knows?

 I need for the redirection to be to pycon.python.org.ar (and we'll
 take care of proper year-by-year redirection from there).

-- 
Marc-Andre Lemburg
Director
Python Software Foundation
http://www.python.org/psf/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] hg.python.org cloning troubles after Sep 13 changes

2014-11-07 Thread M.-A. Lemburg

Just FYI (and for the archives), to perhaps save someone a few minutes:

I've been hitting a problem with hg pull and hg clone on a box recently
and after staring at it for a while, finally found the cause.

Here's what hg printed:

HG-Python/cpython hg pull -u -b 2.7
abort: no suitable response from remote hg!

Not much help.

Using --debug:

orig/HG-Python hg --debug clone ssh://h...@hg.python.org/cpython cpython-2.7
running ssh -C -q -l marc-andre.lemburg -i /home/lemburg/.ssh/id_rsa 
h...@hg.python.org hg -R cpython
serve --stdio
sending hello command
sending between command
abort: no suitable response from remote hg!

Still no help.

Then I remembered that the box was newly setup on Sept 13 and checked
Benjamins email. The server host key had also changed. After removing
the offending key and accepting the new one, things started working
again.

The hg on the box is 1.7.5, so newer version may include a better
error message.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 07 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net

2014-11-06 Thread M.-A. Lemburg

On 06.11.2014 15:39, Brett Cannon wrote:
 What is pythontest.net? Is it something we control, and if so how do we add
 things to it for tests? Did I miss an email on python-dev or
 python-committers about this?

pythontest.net is a domain owned by the PSF and run by Donald
Stufft and Benjamin (I believe).

https://bugs.python.org/issue22650 has the details.

 On Thu Nov 06 2014 at 8:57:22 AM georg.brandl python-check...@python.org
 wrote:
 
 https://hg.python.org/cpython/rev/0af36ea1d010
 changeset:   93417:0af36ea1d010
 branch:  2.7
 parent:  93401:3e8d3c4bc17e
 user:Georg Brandl ge...@python.org
 date:Thu Nov 06 14:37:49 2014 +0100
 summary:
   #22650: test suite: load Unicode test data files from www.pythontest.net

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 06 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-10-24: Released eGenix pyOpenSSL 0.13.5 ...  http://egenix.com/go63

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

2014-10-26 Thread M.-A. Lemburg

On 26.10.2014 00:14, Ned Deily wrote:
 In article m28uk4wxod@valheru.db3l.homeip.net,
  David Bolen db3l@gmail.com wrote:
 
 David Bolen db3l@gmail.com writes:

 which appears to die mid-stream while receiving the manifests.

 So I'm sort of hoping there might be some record server-side as to why
 things are falling apart mid-way.

 Just to follow-up to myself, I get the same same error trying to do a
 clone from my own personal XP machine rather than the buildbot (which
 is a VM).  I've had the issue with hg 1.6.2, 2.5.2 and 3.1.2.

 However, the same clones completely successfully under OSX and Linux.

 So that's sort of strange.
 
 Very interesting!  I had been doing some housekeeping on some of my 
 older OS X build systems over the past few days and I've run into the 
 same problem.  In particular, I am seeing this failure on an OS X 10.5.8 
 system (running in a Fusion VM) which I've used for years and from which 
 I have regularly cloned repos from hg.python.org.  I spent some time 
 yesterday trying to isolate it.  I came to the conclusion that it was 
 independent of the version of OpenSSL (identical failures occurred with 
 the system's ancient Apple 0.9.7 as well as a newly-build 1.0.1j) and 
 independent of the version of hg (at least with two data points, current 
 and a year-old version) and seemingly independent of the network 
 connection.  I was not able to reproduce the failure on the host OS X 
 system (10.10) and I didn't have problems a few days earlier with 
 various other OS X releases (10.6.x through 10.9.x) also running in VMs 
 on the same host.  I stumbled across a workaround for the problem as I 
 was experiencing it:  adding --uncompressed to hg clone eliminated 
 failures.  You can get more info on the hg failures by adding 
 --traceback and --debugger to the clone command.  After spending way too 
 much time on the issue, I was not in the mood to spend more time 
 isolating the problem after finding a workaround but if others are also 
 seeing it, it might be worth doing.  Sigh.
 
   $ hg --version
   Mercurial Distributed SCM (version 3.1.2)
   $ hg clone -U http://hg.python.org/cpython cpython
   real URL is https://hg.python.org/cpython
   requesting all changes
   adding changesets
   adding manifests
   transaction abort!
   rollback completed
   abort: connection ended unexpectedly
   $ hg clone --uncompressed -U https://hg.python.org/cpython cpython
   streaming all changes
   10404 files to transfer, 248 MB of data
   transferred 248 MB in 44.4 seconds (5.58 MB/sec)

If compression is causing the problem, perhaps there's an incompatibility
with the use zlib version between the host and your client system.

hg.python.org was recently updated to a new Ubuntu version.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 26 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-10-24: Released eGenix pyOpenSSL 0.13.5 ...  http://egenix.com/go63

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-10 Thread M.-A. Lemburg

On 10.10.2014 11:26, Larry Hastings wrote:
 
 On 10/10/2014 08:07 AM, Paul Moore wrote:
 On 10 October 2014 01:29, Victor Stinner victor.stin...@gmail.com wrote:
 What about the Python stable ABI? Would it be broken if we use a
 different compiler?

 What about third party Python extensions?

 What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.?
 The key point for me is that any supported build on Windows supports
 the exact same ABI.
 
 Just to make something clear that may not be clear to non-Windows developers: 
 the C library is
 implicitly part of the ABI.  So unless these other compilers bend over 
 backwards to generate code /
 have a C library that behaves *exactly* like MSVC, their ABI will be 
 different, and therefore shared
 libraries compiled with one compiler won't work with the next.  Heck, even 
 shared libraries compiled
 with one version of MSVC won't work with any other version!  (This is 
 something apparently being
 fixed by MSVC 15; apparently they are designing the ABI for forwards 
 compatibility.  Huzzah!)
 
 So if CPython officially said we support MSVC and Compiler X, I worry that 
 we'd have third-party
 modules compiled with either one or the other, leaving users unable to mix 
 and match third-party
 extensions as they do today.  (I want to use library X, which is only 
 available compiled by MSVC. 
 I also want to use library Y, which is only available compiled by Compiler X. 
  What should I do?
 ... install Linux?)
 
 
 Here's my perspective.  Having your code compilable by more compilers is 
 good.  But a maze of
 #ifdefs is bad.  We still have #ifdef's for Borland C--I'd be very surprised 
 if anyone was compiling
 Python 3 with Borland C.  IMO the benefit from supporting other compilers on 
 Windows is negligible,
 but the costs in maintaining these other compilers is tangible.  Or, worse, 
 we accept changes to
 support these other compilers, but the support is incomplete, or goes 
 unmaintained and breaks (and
 nobody notices).
 
 So as a practical matter I think I'd prefer if we continued to only support 
 MSVC.  In fact I'd
 prefer it if we removed support for other Windows compilers, instead asking 
 those maintainers to
 publish their own patches / repos, in the way that Stackless does.

I don't think this is special to the Windows platform. We already
do support quite a few compilers in CPython and for multiple
platforms, so keeping support for e.g. MinGW or adding Intel
C support wouldn't really make much difference in the overall
#ifdef picture ;-)

That said, We do need maintainers for this support, so if there
are no people willing to support these compilers in CPython,
we should use the external port hosting approach for these, IMO.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

2014-09-24 Thread M.-A. Lemburg

On 24.09.2014 03:48, Nick Coghlan wrote:
 On 24 September 2014 03:05, Steve Dower steve.do...@microsoft.com wrote:
 Larry Hastings wrote:

 On 09/19/2014 03:31 PM, Barry Warsaw wrote:
 I think we need a Python 3.5 Release Schedule PEP.

 Just checked it in as PEP 478.  It should show up here in a few minutes:
 http://legacy.python.org/dev/peps/pep-0478/

 Key facts:
 . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 
 sprints.
 . Final release is September 13, 2015, just over a year from now.

 Comments?

 Martin is no longer producing the Windows installers - that task has been 
 handed to me. I'm planning to have a rewritten installer (also in the same 
 repo) that should be easier to modify and maintain, as well as being able to 
 produce alternative packages (such as a Python 3.5 or stdlib merge module, 
 for example), though that doesn't necessarily need to go into the PEP.

 I'm also considering/experimenting with installing into Program Files by 
 default, but I suspect that isn't going to work out yet.

 I'd like to move the Windows versions onto the next release of VC (currently 
 VC 14 until the branding team figures out what to call it). There isn't a 
 promised RTM date for VC 14 yet, so it looks like the best available 
 compiler by Beta 1 will be a Go Live RC. (The Go Live marking basically 
 means we think this is ready for use, but expect a round of minor 
 updates/fixes soon - the compiler is least likely to be updated, my guess is 
 that it'll be Visual Studio UI mostly).

 I personally don't have any qualms about using the RC compiler for Beta 1, 
 and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed 
 this topic that some people were concerned about having the final version 
 available for Python 3.5 Beta.

 So far I've been building regularly with internal versions of VC and haven't 
 been hitting any major issues with Python (OpenSSL has some issues, but I've 
 been filing bugs on both sides so they should be worked out soon enough). My 
 work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for 
 anyone interested.

 For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), 
 but I obviously want to settle on one or the other by Beta. Last time we 
 discussed it, there was strong support for changing compiler, but I have a 
 better idea of the timeline now and it's tighter than I thought...

 Thoughts, anyone?
 
 It's ultimately up to Larry as RM, but I'd personally favour targeting
 the newer compiler and runtime, even with the slight risk of
 potentially needing to slip our schedule. There's also a fair amount
 of wiggle room between the first beta and the first release candidate.

I'd rather be conservative here and wait for another Python release
before switching VC versions. There are a few important questions
that need answers before we can consider a new VC version:

 * Will there be free versions available ?

 * Will those free editions include the 64-bit compilers ?

 * Will those free editions include the optimizing compilers ?

 * Is there a roadmap for how long these free versions will remain
   officially available ?

 * Are there issues compiling 3rd party libraries with it ?

   E.g. the numeric and science stacks, the web stacks,
   the deployment stacks, etc.

 * What license terms will the new version have ?

   E.g. GPL compatibility issues, weird exceptions,

 * What will the pricing structure look like ?

   While core devs will get free MSDN licenses, most other 3rd party
   providers will have to buy licenses for the compiler, unless
   they can use the free versions.

An alternative would be targeting VC13 instead of VC14, in case it
has good answers for the above questions. It's been around for a
year now, so there should be more experience available with this
version.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 24 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-09-27: PyDDF Sprint 2014 ...   3 days to go
2014-09-30: Python Meeting Duesseldorf ...  6 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

2014-09-24 Thread M.-A. Lemburg

Thanks for the insights, Steve.

More below...

On 24.09.2014 18:52, Steve Dower wrote:
 M.-A. Lemburg wrote:

 I'd rather be conservative here and wait for another Python release before
 switching VC versions. There are a few important questions that need answers
 before we can consider a new VC version:

 * Will there be free versions available ?

 * Will those free editions include the 64-bit compilers ?

 * Will those free editions include the optimizing compilers ?

 * Is there a roadmap for how long these free versions will remain
 officially available ?

 * Are there issues compiling 3rd party libraries with it ?

 E.g. the numeric and science stacks, the web stacks,
 the deployment stacks, etc.

 * What license terms will the new version have ?

 E.g. GPL compatibility issues, weird exceptions,

 * What will the pricing structure look like ?

 While core devs will get free MSDN licenses, most other 3rd party
 providers will have to buy licenses for the compiler, unless
 they can use the free versions.

 An alternative would be targeting VC13 instead of VC14, in case it has good
 answers for the above questions. It's been around for a year now, so there
 should be more experience available with this version.
 
 (Nit - it's actually VC12 a.k.a. Visual Studio 2013 - VC13 was skipped. 
 This is what happens when you have separate engineering and marketing teams 
 :) )

Ah, ok :-)

 I don't have good answers to all of these yet, but none of them are going to 
 be any worse than for VC12. I've forwarded these questions to the people on 
 the VC team who do get to choose the answers, and while I'm not expecting to 
 hear specifics back from them, they are at least aware of the concerns and 
 how important their product is to our community.
 
 There will be free versions available, but I don't know what format they'll 
 be in. Those free editions should include identical compilers to the paid 
 ones - the cases where that hasn't been true have been bugs or due to 
 assumptions that were proven to be incorrect.
 
 The main improvement in this version is that all versions from VC14 should be 
 binary compatible, and so there will always be a free compiler, but it may be 
 VC15/16/etc. and not VC14.

That's good news.

 There are certainly issues with 3rd party libraries, largely because all 
 projects have a tendency to take dependencies on compiler/library internals. 
 OpenSSL, for example, redefines the stdout/in/err macros based on the VC 
 version, but the new definitions are no longer valid with VC14, and so they 
 are fixing that. Python itself has a few issues that I have already fixed in 
 my branch. There will certainly be other issues, but an advantage of starting 
 early is that bugs in the compiler itself can be fixed in the compiler.
 
 The license should not change significantly from previous versions. GPL 
 incompatibilities are because the GPL wants to be incompatible with licenses 
 based on different ideologies - AFAIK there's never been anything in the VC 
 licenses preventing whatever redistribution license you like.

As example: there once was a special clause which explicitly
disallowed Excluded License[s] to be used together with
VC redistibutable files. I think this is no longer the case, but
there may be new things in the EULAs.

 Part of my improvements to /PCBuild will help avoid the need for Visual 
 Studio entirely, but the free versions should always be sufficient for 
 building and debugging. I have no insight or control over the pricing 
 structure.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 24 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-09-27: PyDDF Sprint 2014 ...   3 days to go
2014-09-30: Python Meeting Duesseldorf ...  6 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

2014-09-08 Thread M.-A. Lemburg

On 01.09.2014 10:09, Nick Coghlan wrote:
 On 1 September 2014 17:13, Christian Heimes christ...@python.org wrote:
 On 01.09.2014 08:44, Nick Coghlan wrote:
 Yes, it would have exactly the same security failure modes as
 sitecustomize, except it would only fire if the application
 imported the ssl module.

 The -S and -I switches would need to disable the implied
 sslcustomize, just as they disable import site.

 A malicious package can already play havoc with your installation with
 a custom ssl module. If somebody is able to sneak in a ssl.py then you
 are screwed anyway. sslcustomize is not going to make the situation worse.
 
 That's not quite true - we're fairly careful about putting the
 standard library before userspace directories, so aside from the
 current directory problem, shadowing ssl itself can be tricky to
 arrange.

It's really easy to modify sys.modules to override any module
that has already been loaded or add new ones bypassing sys.path
entirely, so the sys.path layout doesn't provide any protection
against such hacks.

If you gain access to one of the dirs on sys.path, you can play
such tricks in sitecustomize.py. Any 3rd party package can do
the same.

We'd have to add digital API signatures to the ssl module to
prevent such stunts :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...11 days to go
2014-09-27: PyDDF Sprint 2014 ...  19 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] surrogatepass - she's a witch, burn 'er! [was: Cleaning up ...]

On 30.08.2014 01:37, Greg Ewing wrote:
 M.-A. Lemburg wrote:
 we needed
 a way to make sure that Python 3 also optionally supports working
 with lone surrogates in such UTF-8 streams (nowadays called CESU-8:
 http://en.wikipedia.org/wiki/CESU-8).
 
 I don't think CESU-8 is the same thing. According to the wiki
 page, CESU-8 *requires* all code points above 0x to be split
 into surrogate pairs before encoding. It also doesn't say that
 lone surrogates are valid -- it doesn't mention lone surrogates
 at all, only pairs. Neither does the linked technical report.
 
 The technical report also says that CESU-8 forbids any UTF-8
 sequences of more than three bytes, so it's definitely not
 UTF-8 plus lone surrogates.

You're right, it's not the same as UTF-8 plus lone surrogates.

CESU-8 does encode surrogates as individual code points using
the UTF-8 encoding, which is what probably caused it to be
mentioned in discussions when talking about having UTF-8 streams
do the same for lone surrogates.

So let's call the encoding UTF-8-py so that everyone knows
what we're talking about :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 30 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go
2014-09-27: PyDDF Sprint 2014 ...  28 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

On 30.08.2014 04:44, Alex Gaynor wrote:
 Thanks for the rapid feedback everyone!
 
 I want to summarize the action items and discussion points that have come up 
 so
 far:
 
 To add to the PEP:
 
 * Emit a warning in 3.4.next for cases that would raise a Exception in 3.5
 * Clearly state that the existing OpenSSL environment variables will be
   respected for setting the trust root

I'd also suggest to compile Python with OPENSSL_LOAD_CONF, since that
causes OpenSSL to read the global openssl.cnf file for additional
configuration.

 Discussion points:
 
 * Disabling verification entirely externally to the program, through a CLI 
 flag
   or environment variable. I'm pretty down on this idea, the problem you hit 
 is
   that it's a pretty blunt instrument to swing, and it's almost impossible to
   imagine it not hitting things it shouldn't; it's far too likely to be used 
 in
   applications that make two sets of outbound connections: 1) to some internal
   service which you want to disable verification on, and 2) some external
   service which needs strong validation. A global flag causes the latter to
   fail silently when subjected to a MITM attack, and that's exactly what we're
   trying to avoid. It also makes things much harder for library authors: I
   write an API client for some API, and make TLS connections to it. I want
   those to be verified by default. I can't even rely on the httplib defaults,
   because someone might disable them from the outside.

The reasoning here is the same as for hash randomization. There
are cases where you want to test your application using self-signed
certificates which don't validate against the system CA root list.

In those cases, you do know what you're doing. The test would fail
otherwise and the reason is not a bug in your code, it's just
the fact that the environment you're running it in is a test
environment.

Ideally, all applications should give you this choice, but this is
unlikely to happen, so it's good to be able to change the Python
default, since with the proposed change, most applications will
probably continue to use the Python defaults as they do now.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 30 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go
2014-09-27: PyDDF Sprint 2014 ...  28 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

On 30.08.2014 12:40, Antoine Pitrou wrote:
 On Sat, 30 Aug 2014 12:19:11 +0200
 M.-A. Lemburg m...@egenix.com wrote:
 To add to the PEP:

 * Emit a warning in 3.4.next for cases that would raise a Exception in 3.5
 * Clearly state that the existing OpenSSL environment variables will be
   respected for setting the trust root

 I'd also suggest to compile Python with OPENSSL_LOAD_CONF, since that
 causes OpenSSL to read the global openssl.cnf file for additional
 configuration.
 
 Python links against OpenSSL as a shared library, not statically. It's
 unlikely that setting a compile constant inside Python would affect
 OpenSSL at all.

The change is to the OpenSSL API, not the OpenSSL lib. By setting
the variable you enable a few special calls to the config loader
functions in OpenSSL when calling the initializer it:

https://www.openssl.org/docs/crypto/OPENSSL_config.html

 Discussion points:

 * Disabling verification entirely externally to the program, through a CLI 
 flag
   or environment variable. I'm pretty down on this idea, the problem you 
 hit is
   that it's a pretty blunt instrument to swing, and it's almost impossible 
 to
   imagine it not hitting things it shouldn't; it's far too likely to be 
 used in
   applications that make two sets of outbound connections: 1) to some 
 internal
   service which you want to disable verification on, and 2) some external
   service which needs strong validation. A global flag causes the latter to
   fail silently when subjected to a MITM attack, and that's exactly what 
 we're
   trying to avoid. It also makes things much harder for library authors: I
   write an API client for some API, and make TLS connections to it. I want
   those to be verified by default. I can't even rely on the httplib 
 defaults,
   because someone might disable them from the outside.

 The reasoning here is the same as for hash randomization. There
 are cases where you want to test your application using self-signed
 certificates which don't validate against the system CA root list.
 
 That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE
 env vars (or, better, by specific settings *inside* the application).
 
 I'm against multiplying environment variables, as it makes it more
 difficult to assess the actual security of a setting. The danger of an
 ill-secure setting is much more severe than with hash randomization.

You have a point there. So how about just a python run-time switch
and no env var ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 30 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go
2014-09-27: PyDDF Sprint 2014 ...  28 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

On 30.08.2014 12:55, Antoine Pitrou wrote:
 On Sat, 30 Aug 2014 12:46:47 +0200
 M.-A. Lemburg m...@egenix.com wrote:
 The change is to the OpenSSL API, not the OpenSSL lib. By setting
 the variable you enable a few special calls to the config loader
 functions in OpenSSL when calling the initializer it:

 https://www.openssl.org/docs/crypto/OPENSSL_config.html
 
 Ah, ok. Do you have experience with openssl.cnf? Apparently, it is
 meant for offline tools such as certificate generation, I am not sure
 how it could impact certification validation.

I'm still exploring this: the OpenSSL documentation is, well,
less than complete on these things, so searching mailing lists
and reading source code appears to be the only reasonable way
to figure out what is possible and what not.

The openssl.cnf config file is indeed mostly used by the various
openssl subcommands (e.g. req and ca), but it can also be used to
configuring engines and my hope is that configuration of
e.g. default certificate stores also becomes possible.

One of the engines can tap into the Windows certificate store,
for example.

 That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE
 env vars (or, better, by specific settings *inside* the application).

 I'm against multiplying environment variables, as it makes it more
 difficult to assess the actual security of a setting. The danger of an
 ill-secure setting is much more severe than with hash randomization.

 You have a point there. So how about just a python run-time switch
 and no env var ?
 
 Well, why not, but does it have a value over letting the code properly
 configure their SSLContext?

Yes, because when Python changes the default to be validating
and more secure, application developers will do the same as
they do now: simply use the defaults ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 30 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go
2014-09-27: PyDDF Sprint 2014 ...  28 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

On 30.08.2014 15:32, R. David Murray wrote:
 On Sat, 30 Aug 2014 14:03:57 +0200, M.-A. Lemburg m...@egenix.com wrote:
 On 30.08.2014 12:55, Antoine Pitrou wrote:
 On Sat, 30 Aug 2014 12:46:47 +0200
 M.-A. Lemburg m...@egenix.com wrote:
 That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE
 env vars (or, better, by specific settings *inside* the application).

 I'm against multiplying environment variables, as it makes it more
 difficult to assess the actual security of a setting. The danger of an
 ill-secure setting is much more severe than with hash randomization.

 You have a point there. So how about just a python run-time switch
 and no env var ?

 Well, why not, but does it have a value over letting the code properly
 configure their SSLContext?

 Yes, because when Python changes the default to be validating
 and more secure, application developers will do the same as
 they do now: simply use the defaults ;-)
 
 But neither of those addresses the articulated use case: someone *using*
 a program implemented in python that does not itself provide a way to
 disable the new default security (because it is *new*).  Only an
 environment variable will do that.
 
 Since the environment variable is opt-in, I think the consenting
 adults argument applies to Alex's demure about multiple connections.
 It could still emit the warnings.

That would be a possibility as well, yes.

I'd just like to see a way to say: I know what I'm doing
and I'm not in the mood to configure my own CA list, so
please go ahead and accept whatever certs you find --
much like what --no-check-certificate does for wget.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 30 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go
2014-09-27: PyDDF Sprint 2014 ...  28 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] surrogatepass - she's a witch, burn 'er! [was: Cleaning up ...]

On 29.08.2014 02:41, Stephen J. Turnbull wrote:
 In the process of booking up for my other post in this thread, I
 noticed the 'surrogatepass' handler.
 
 Is there a real use case for the 'surrogatepass' error handler?  It
 seems like a horrible break in the abstraction.  IMHO, if there's a
 need, the application should handle this.  Python shouldn't provide
 it on encoding as the resulting streams are not Unicode conformant,
 nor on decoding UTF-16, as conversion of surrogate pairs is a
 requirement of all Unicode versions since about 1995.

This error handler allows applications to reactivate the Python 2
style behavior of the UTF codecs in Python 3, which allow reading
lone surrogates on input.

Since Python allows working with lone surrogates in Unicode (they
are valid code points) and we're using UTF-8 for marshal, we needed
a way to make sure that Python 3 also optionally supports working
with lone surrogates in such UTF-8 streams (nowadays called CESU-8:
http://en.wikipedia.org/wiki/CESU-8).

See

http://bugs.python.org/issue3672
http://bugs.python.org/issue12892

for discussions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go
2014-09-27: PyDDF Sprint 2014 ...  29 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] surrogatepass - she's a witch, burn 'er! [was: Cleaning up ...]

On 29.08.2014 13:22, Isaac Morland wrote:
 On Fri, 29 Aug 2014, M.-A. Lemburg wrote:
 
 On 29.08.2014 02:41, Stephen J. Turnbull wrote:
 Since Python allows working with lone surrogates in Unicode (they
 are valid code points) and we're using UTF-8 for marshal, we needed
 a way to make sure that Python 3 also optionally supports working
 with lone surrogates in such UTF-8 streams (nowadays called CESU-8:
 http://en.wikipedia.org/wiki/CESU-8).
 
 If I want that wouldn't I specify cesu-8 as the encoding?
 
 i.e., instead of .decode ('utf-8') I would use .decode ('cesu-8').  Right 
 now, trying this I get
 that cesu-8 is an unknown encoding but that could be changed without 
 affecting the behaviour of the
 utf-8 codec.

Why write a new codec that's almost identical to the utf-8 codec,
if you can get the same functionality by explicitly using a
special error handler ?

From a maintenance POV that does not sound like a good approach.

 It seems to me that .decode ('utf-8') should decode exactly and only valid 
 utf-8, including the
 non-use of surrogate pairs as an intermediate encoding step.

It does in Python 3.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go
2014-09-27: PyDDF Sprint 2014 ...  29 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

On 29.08.2014 21:47, Alex Gaynor wrote:
 Hi all,
 
 I've just submitted PEP 476, on enabling certificate validation by default for
 HTTPS clients in Python. Please have a look and let me know what you think.
 
 PEP text follows.

Thanks for the PEP. I think this is generally a good idea,
but some important parts are missing from the PEP:

 * transition plan:

   I think starting with warnings in Python 3.5 and going
   for exceptions in 3.6 would make a good transition

   Going straight for exceptions in 3.5 is not in line with
   our normal procedures for backwards incompatible changes.

 * configuration:

   It would be good to be able to switch this on or off
   without having to change the code, e.g. via a command
   line switch and environment variable; perhaps even
   controlling whether or not to raise an exception or
   warning.

 * choice of trusted certificate:

   Instead of hard wiring using the system CA roots into
   Python it would be good to just make this default and
   permit the user to point Python to a different set of
   CA roots.

   This would enable using self signed certs more easily.
   Since these are often used for tests, demos and education,
   I think it's important to allow having more control of
   the trusted certs.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go
2014-09-27: PyDDF Sprint 2014 ...  29 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

On 29.08.2014 23:11, Donald Stufft wrote:
 
 Sorry I was on my phone and didn’t get to fully reply to this.
 
 On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg m...@egenix.com wrote:

 On 29.08.2014 21:47, Alex Gaynor wrote:
 Hi all,

 I've just submitted PEP 476, on enabling certificate validation by default 
 for
 HTTPS clients in Python. Please have a look and let me know what you think.

 PEP text follows.

 Thanks for the PEP. I think this is generally a good idea,
 but some important parts are missing from the PEP:

 * transition plan:

   I think starting with warnings in Python 3.5 and going
   for exceptions in 3.6 would make a good transition

   Going straight for exceptions in 3.5 is not in line with
   our normal procedures for backwards incompatible changes.
 
 As far as a transition plan, I think that this is an important
 enough thing to have an accelerated process. If we need
 to provide a warning than let’s add it to the next 3.4 otherwise
 it’s going to be 2.5+ years until we stop being unsafe by
 default.

Fine with me; we're still early in the Python 3.4
patch level releases.

 Another problem with this is that I don’t think it’s actually
 possible to do. Python itself isn’t validating the TLS certificates,
 OpenSSL is doing that. To my knowledge OpenSSL doesn’t
 have a way to say “please validate these certificates and if
 they don’t validate go ahead and keep going and just let me
 get a warning from it”. It’s a 3 way switch, no validation, validation
 if a certificate is provided, and validation always.
 
 Now that’s strictly for the “verify the certificate chain” portion,
 the hostname verification is done entirely on our end and we
 could do something there… but I’m not sure it makes sense
 to do so if we can’t do it for invalid certificates too.

OpenSSL provides a callback for certificate validation,
so it is possible to issue a warning and continue with
accepting the certificate.

 * configuration:

   It would be good to be able to switch this on or off
   without having to change the code, e.g. via a command
   line switch and environment variable; perhaps even
   controlling whether or not to raise an exception or
   warning.
 
 I’m on the fence about this, if someone provides a certificate
 that we can validate against (which can be done without
 touching the code) then the only thing that really can’t be
 “fixed” without touching the code is if someone has a certificate
 that is otherwise invalid (expired, not yet valid, wrong hostname,
 etc). I’d say if I was voting on this particular thing I’d be -0, I’d
 rather it didn’t exist but I wouldn’t cry too much if it did.

If you're testing code or trying out some new stuff, you
don't want to get a valid cert first, but instead go ahead
with a self signed one. That's the use case.

 * choice of trusted certificate:

   Instead of hard wiring using the system CA roots into
   Python it would be good to just make this default and
   permit the user to point Python to a different set of
   CA roots.

   This would enable using self signed certs more easily.
   Since these are often used for tests, demos and education,
   I think it's important to allow having more control of
   the trusted certs.
 
 
 Like my other email said, the Python API has everything needed
 to easily specify your own CA roots and/or disable the validations.
 The OpenSSL library also allows you to specify either a directory
 or a file to change the root certificates without code changes. The
 only real problems with the APIs are that the default is bad and
 an unrelated thing where you can’t pass in an in memory certificate.

Are you sure that's possible ? Python doesn't load the
openssl.cnf file and the SSL_CERT_FILE, SSL_CERT_DIR env
vars only work for the openssl command line binary, AFAIK.

In any case, Python will have to tap into the OS CA root
provider using special code and this code could easily be
made to check other dirs/files as well.

The point is that it should be possible to change this default
at the Python level, without needing application code changes.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ...   http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go
2014-09-27: PyDDF Sprint 2014 ...  29 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact

Re: [Python-Dev] Python 2.7 patch levels turning two digit

2014-06-23 Thread M.-A. Lemburg

On 23.06.2014 18:09, Donald Stufft wrote:
 
 On Jun 23, 2014, at 2:09 AM, Martin v. Löwis mar...@v.loewis.de wrote:
 

 * Should we make use of the potential breakage with 2.7.10
   to introduce a new Windows compiler version for Python 2.7 ?

 Assuming it is a good idea to continue producing Windows binaries
 for 2.7, I think it would be a bad idea to switch compilers. It will
 cause severe breakage of 2.7 installations, much more problematic
 than switching to two-digit version numbers.
 
 I agree with this, we’ve just finally started getting things to the point 
 where
 it makes a lot of sense for binary distributions for Windows. Breaking all
 of them on 2.7 would be very bad.

Not sure what you mean. We've had binary wininst distributions
for Windows for more than a decade, and egg and msi distributions
for 8 years :-)

But without access to the VS 2008 compiler that is needed to
compile those extensions, it will become increasingly difficult
for package authors to provide such binary packages, so we have to
ask ourselves:

What's worse: breaking old Windows binaries for Python 2.7
or not having updated and new Windows binaries for Python 2.7
at all in a few years ?

Switching to a newer compiler will make things easier for everyone
and we'd see more binary packages for Windows again.

Given that Python 2.7 support was extended for another 5 years at the
recent Python Language Summit to 2020, we have to face this
breakage sooner or later anyway. Extended support for VS 2008
will end in 2018 (but then: Python developers usually don't have
extended support contracts with MS). Service pack support has already
ended in 2009.

Depending on how you see it, using such an old compiler also
poses security risks. The last security update for VS 2008 dates
back to 2011 (http://support.microsoft.com/kb/2538243).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 23 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ...   http://egenix.com/go58
2014-07-02: Python Meeting Duesseldorf ...  9 days to go
2014-07-21: EuroPython 2014, Berlin, Germany ...   28 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

2014-06-23 Thread M.-A. Lemburg

On 23.06.2014 22:20, Donald Stufft wrote:
 
 On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 23.06.2014 18:09, Donald Stufft wrote:

 On Jun 23, 2014, at 2:09 AM, Martin v. Löwis mar...@v.loewis.de wrote:


 * Should we make use of the potential breakage with 2.7.10
  to introduce a new Windows compiler version for Python 2.7 ?

 Assuming it is a good idea to continue producing Windows binaries
 for 2.7, I think it would be a bad idea to switch compilers. It will
 cause severe breakage of 2.7 installations, much more problematic
 than switching to two-digit version numbers.

 I agree with this, we’ve just finally started getting things to the point 
 where
 it makes a lot of sense for binary distributions for Windows. Breaking all
 of them on 2.7 would be very bad.
 
 Err, sorry that “We” was with my pip hat on.
 

 Not sure what you mean. We've had binary wininst distributions
 for Windows for more than a decade, and egg and msi distributions
 for 8 years :-)
 
 Nonetheless, changing the compiler will not only break pip, but every
 automated installer tool (easy_install, buildout) that i’m aware of. The
 blow back for binary installation is going to be huge I think.

 But without access to the VS 2008 compiler that is needed to
 compile those extensions, it will become increasingly difficult
 for package authors to provide such binary packages, so we have to
 ask ourselves:

 What's worse: breaking old Windows binaries for Python 2.7
 or not having updated and new Windows binaries for Python 2.7
 at all in a few years ?
 
 At the risk of getting Guido to post his slide again, I still think the
 solution to the old compiler is to just roll a 2.8 with minimal changes.
 It could even be a good place to move to the ssl backport changes
 too since they were the riskier set of changes in PEP466.
 
 But either way, if a compiler does change in a 2.7 release we’ll need
 to update a lot of tooling to cope with that, so any plan to do that should
 include that and a timeline for adoption of that.

Sure, and we'd need to hash out possible solutions to minimize
breakage, but first we'll have to see whether we want to consider
this step or not.


BTW: It's strange that I'm arguing for breaking things. I'm usually
on the other side of such arguments :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 2.7 patch levels turning two digit

2014-06-21 Thread M.-A. Lemburg

With PEP 466 and the constant flow of OpenSSL security fixes
which are currently being handled via Python patch level releases,
we will soon reach 2.7.10 and quickly go beyond that (also see
http://bugs.python.org/issue21308).

This opens up a potential backwards incompatibility with existing
tools that assume the Python release version number to use the
x.y.z single digit approach, e.g. code that uses sys.version[:5]
for the Python version or relies on the lexicographic ordering
of the version string (sys.version  '2.7.2').

Some questions we should probably ask ourselves (I've added my
thoughts inline):

 * Is it a good strategy to ship to Python releases for every
   single OpenSSL security release or is there a better way to
   handle these 3rd party issues ?

   I think we should link to the OpenSSL libs dynamically rather
   than statically in Python 2.7 for Windows so that it's possible
   to provide drop-in updates for such issues.

 * Should we try to avoid two digit patch level release numbers
   by using some other mechanism such as e.g. a release date
   after 2.7.9 ?

   Grepping through our code, this will introduce some breakage,
   but not much. Most older code branches on minor versions,
   not patch levels. More recent code uses sys.python_info so
   is not affected.

 * Should we make use of the potential breakage with 2.7.10
   to introduce a new Windows compiler version for Python 2.7 ?

   I think this would be a good chance to update the compiler
   to whatever we use for Python 3 at the time.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 21 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ...   http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ...  http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 11 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

2014-06-21 Thread M.-A. Lemburg

On 21.06.2014 12:51, Nick Coghlan wrote:
 On 21 June 2014 20:27, M.-A. Lemburg m...@egenix.com wrote:
 With PEP 466 and the constant flow of OpenSSL security fixes
 which are currently being handled via Python patch level releases,
 we will soon reach 2.7.10 and quickly go beyond that (also see
 http://bugs.python.org/issue21308).

 This opens up a potential backwards incompatibility with existing
 tools that assume the Python release version number to use the
 x.y.z single digit approach, e.g. code that uses sys.version[:5]
 for the Python version or relies on the lexicographic ordering
 of the version string (sys.version  '2.7.2').
 
 Such code has an easy fix available, though, as sys.version_info has
 existed since 2.0, and handles two digit micro releases just fine. The
 docs for sys.version also have this explicit disclaimer: Do not
 extract version information out of it, rather, use version_info and
 the functions provided by the platform module.

I don't think that's a good argument. Of course, there are
better ways to figure out the version number, but fact is,
existing code, even in the stdlib, does use and parse
the sys.version string version.

During Python's lifetime, we've always avoided two digit
version numbers, so people have been relying on this, even
if it was never (AFAIK) documented anywhere.

 Making it harder to tell whether or not someone's Python installation
 is affected by an OpenSSL CVE is also an undesirable outcome. On a
 Linux distro, folks will check the distro package database directly
 for the OpenSSL version, but on Windows, no such centralised audit
 mechanism is available by default. With OpenSSL statically linked,
 Python versions can just be mapped to OpenSSL versions (so, for
 example, 2.7.7 has 1.0.1g)

I have to disagree here as well :-)

If people cannot upgrade to a higher patch level for whatever
reason (say a patch level release introduced some other bugs),
but still need to upgrade to the current OpenSSL version, they'd
be stuck if we continue to bind the Python version number to
some OpenSSL release version.

We should definitely make it possible to address OpenSSL
bugs without having to upgrade Python and it's not hard to
do: just replace the static binding with dynamic binding
and include the two OpenSSL DLLs with the Windows installer.

People can then drop in new versions of those DLLs
as needed, without having the core devs do a complete
new release every time someone finds a new problem those
libs. Security libs simply have a much higher release
rate (if they are well maintained) than most other
software.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 21 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ...   http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ...  http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 11 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

2014-06-21 Thread M.-A. Lemburg

On 21.06.2014 22:34, Chris Angelico wrote:
 On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg m...@egenix.com wrote:
 On 21.06.2014 12:51, Nick Coghlan wrote:
 Such code has an easy fix available, though, as sys.version_info has
 existed since 2.0, and handles two digit micro releases just fine. The
 docs for sys.version also have this explicit disclaimer: Do not
 extract version information out of it, rather, use version_info and
 the functions provided by the platform module.

 I don't think that's a good argument. Of course, there are
 better ways to figure out the version number, but fact is,
 existing code, even in the stdlib, does use and parse
 the sys.version string version.

 During Python's lifetime, we've always avoided two digit
 version numbers, so people have been relying on this, even
 if it was never (AFAIK) documented anywhere.
 
 It's going to be a broken-code-breaking change that's introduced in a
 point release, but since PEP 404 implicitly says that there won't be a
 2.10.0, there's no way around that. Although actually, a glance at the
 stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than
 2.7.10 would break - there are places where sys.version[:3] is used
 (or equivalents like ... %.3s ... % sys.version), or a whole-string
 comparison is done against a two-part version string (eg: sys.version
 = 2.6), and at least one place that checks sys.version[0] for the
 major version number, but I didn't find any that look at
 sys.version[:5] or equivalent. Everything that cares about the
 three-part version number seems to either look at
 sys.version.split()[0] or sys.version_info. Do you know where this
 problematic code is?
 
 I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy
 system, for what it's worth.

There are no places in the stdlib that parse sys.version in a
way that would break wtih 2.7.10, AFAIK. I was just referring
to the statement that Nick quoted. sys.version *is* used for
parsing the Python version or using parts of it to build
e.g. filenames and that's really no surprise.

That said, and I also included this in my answers to the questions
that Nick removed in his reply, I don't think that a lot of
code would be affected by this. I do believe that we can use
this potential breakage as a chance for improvement. See the last
question (listed here again)...

1. Is it a good strategy to ship to Python releases for every
   single OpenSSL security release or is there a better way to
   handle these 3rd party issues ?

2. Should we try to avoid two digit patch level release numbers
   by using some other mechanism such as e.g. a release date
   after 2.7.9 ?

3. Should we make use of the potential breakage with 2.7.10
   to introduce a new Windows compiler version for Python 2.7 ?

My answers to these are: 1. We should use dynamic linking
instead and not let OpenSSL bugs trigger Python releases; 2.
It's not a big problem; 3. Yes, please, since it is difficult
for people to develop and debug their extensions with a
2008 compiler, when the rest of the world has long moved on.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 21 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ...   http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ...  http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 11 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 21671: CVE-2014-0224 OpenSSL upgrade to 1.0.1h on Windows required

2014-06-17 Thread M.-A. Lemburg

On 17.06.2014 20:27, Steve Dower wrote:
 Yates, Andy (CS Houston, TX) wrote:
 Python Dev,
 Andy here. I have a Windows product based on Python and I'm getting hammered 
 to
 release a version that includes the fix in OpenSSL 1.0.1h. My product is 
 built
 on a Windows system using Python installed from the standard Python 
 installer at
 Python.org. I would be grateful if I could get some advice on my options. 
 Will
 Python.org be releasing a Windows installer with the fix any time soon or 
 will
 it be at the next scheduled release in November? If it is November, there's 
 no
 way I can wait that long. Now what? Would it be best to build my own Python? 
 Is
 it possible to drop in new OpenSSL versions on Windows without rebuilding
 Python? Looking for some guidance on how to handle these OpenSSL issues on
 Windows.
 
 You'll only need to rebuild the _ssl and _hashlib extension modules with the 
 new OpenSSL version. The easiest way to do this is to build from source 
 (which has already been updated for 1.0.1h if you use the externals scripts 
 in Tools\buildbot), and you should just be able to drop _ssl.pyd and 
 _hashlib.pyd on top of a normal install.
 
 Aside: I wonder if it's worth changing to dynamically linking to OpenSSL? It 
 would make this kind of in-place upgrade easier when people need to do it. 
 Any thoughts? (Does OpenSSL even support it?)

Yes, no problem at all, but you'd still have to either do a new release
every time a new OpenSSL problem is found (don't think that's an
option for Python) or provide new compiled versions
compatible with the Python modules needing the OpenSSL libs or
instructions on how to build these.

Note that the hash routines are rarely affected by these OpenSSL
bugs. They usually only affect the SSL/TLS protocol parts.

Alternatively, you could make use of our pyOpenSSL distribution,
which includes pyOpenSSL and the OpenSSL libs (also for Windows):

http://www.egenix.com/products/python/pyOpenSSL/

We created this to address the problem of having to update
OpenSSL rather often. It doesn't support Python 3 yet, but
on the plus side, you do get OpenSSL libs which are compiled
with the same compiler versions used for the Python.org
installers.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 21671: CVE-2014-0224 OpenSSL upgrade to 1.0.1h on Windows required

2014-06-17 Thread M.-A. Lemburg

On 17.06.2014 22:36, Antoine Pitrou wrote:
 Le 17/06/2014 14:55, M.-A. Lemburg a écrit :

 Alternatively, you could make use of our pyOpenSSL distribution,
 which includes pyOpenSSL and the OpenSSL libs (also for Windows):

 http://www.egenix.com/products/python/pyOpenSSL/

 We created this to address the problem of having to update
 OpenSSL rather often.
 
 This is very nice, but does it also upgrade the OpenSSL version used by the 
 _ssl and _hashlib modules?

On Unix, tt will if you load pyOpenSSL before importing _ssl or
_hashlib (and those modules are built as shared libs).

Alternatively, you can set LD_LIBRARY_PATH to lib/python2.7/OpenSSL
to have the system linker use the embedded libs before starting
Python. Then it will always use the up-to-date libs.

On Windows, this won't work, because _ssl and _hashlib are
statically linked against the OpenSSL libs.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ...   http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ...  http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 15 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler

2014-06-06 Thread M.-A. Lemburg

On 06.06.2014 20:25, Brian Curtin wrote:
 On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico ros...@gmail.com wrote:
 On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower steve.do...@microsoft.com 
 wrote:
 Chris Angelico wrote:
 On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower steve.do...@microsoft.com 
 wrote:
 What this means for Python is that C extensions for Python 3.5 and later 
 can be built using any version of MSVC from 14.0 and later.

 Oh, if only this had been available for 2.7!! Actually... this means that 
 14.0 would be a good target for a compiler change for 2.7.x, if such a 
 change is ever acceptable.

 Maybe, but I doubt it will ever be acceptable :)

 Well, there were discussions. Since Python 2.7's support is far
 exceeding the Microsoft promise of support for the compiler it was
 built on, there's going to be a problem, one way or the other. I don't
 know how that's going to end up being resolved.
 
 We're going to have to change it at some point, otherwise we're going
 to have people in 2018 scrambling to find VS2008, which will be 35
 versions too old by then. No matter what we do here, we're going to
 have a tough PR situation, but we have to make something workable. I'd
 rather cause a hassle than outright kill extensions.
 
 I would probably prefer we aim for VS 14 for 3.5, and then explore
 making the same change for the 2.7.x release that comes after 3.5.0
 comes out. Lessons learned and all that.

Are you sure that's an option ? Changing the compiler the stock
Python from python.org is built with will most likely render
existing Python extensions built for 2.7.x with x  (release that comes
after 3.5.0) broken, so users and installation tools will end up
having to pay close attention to the patch level version of Python
they are using... which is something we wanted to avoid after
we ran into this situation with 1.5.1 and 1.5.2 a few years ago.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 06 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-05-28: Released mxODBC.Connect 2.1.0 ... http://egenix.com/go56
2014-07-02: Python Meeting Duesseldorf ... 26 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler

2014-06-06 Thread M.-A. Lemburg

On 06.06.2014 20:49, Brian Curtin wrote:
 On Fri, Jun 6, 2014 at 10:41 PM, M.-A. Lemburg m...@egenix.com wrote:
 On 06.06.2014 20:25, Brian Curtin wrote:
 On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico ros...@gmail.com wrote:
 On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower steve.do...@microsoft.com 
 wrote:
 Chris Angelico wrote:
 On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower steve.do...@microsoft.com 
 wrote:
 What this means for Python is that C extensions for Python 3.5 and 
 later can be built using any version of MSVC from 14.0 and later.

 Oh, if only this had been available for 2.7!! Actually... this means 
 that 14.0 would be a good target for a compiler change for 2.7.x, if 
 such a change is ever acceptable.

 Maybe, but I doubt it will ever be acceptable :)

 Well, there were discussions. Since Python 2.7's support is far
 exceeding the Microsoft promise of support for the compiler it was
 built on, there's going to be a problem, one way or the other. I don't
 know how that's going to end up being resolved.

 We're going to have to change it at some point, otherwise we're going
 to have people in 2018 scrambling to find VS2008, which will be 35
 versions too old by then. No matter what we do here, we're going to
 have a tough PR situation, but we have to make something workable. I'd
 rather cause a hassle than outright kill extensions.

 I would probably prefer we aim for VS 14 for 3.5, and then explore
 making the same change for the 2.7.x release that comes after 3.5.0
 comes out. Lessons learned and all that.

 Are you sure that's an option ? Changing the compiler the stock
 Python from python.org is built with will most likely render
 existing Python extensions built for 2.7.x with x  (release that comes
 after 3.5.0) broken, so users and installation tools will end up
 having to pay close attention to the patch level version of Python
 they are using... which is something we wanted to avoid after
 we ran into this situation with 1.5.1 and 1.5.2 a few years ago.
 
 None of the options are particularly good, but yes, I think that's an
 option we have to consider. We're supporting 2.7.x for 6 more years on
 a compiler that is already 6 years old. Something less than awesome
 for everyone involved is going to have to happen to make that
 possible.

Perhaps we could combine this with the breakage that a Python 2.7.10
would introduce due to the two digit patch level release version ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 06 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-05-28: Released mxODBC.Connect 2.1.0 ... http://egenix.com/go56
2014-07-02: Python Meeting Duesseldorf ... 26 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

2014-05-09 Thread M.-A. Lemburg

On 08.05.2014 23:22, Donald Stufft wrote:
 
 On a personal note, I'm uncomfortable with the way this change is
 perceived as a case of *pip* enforcing a behaviour that the pip
 developers feel should be required. I actually don't like this change
 particularly. So having pip implement the behaviour required by that
 PEP is to me simply a case of compliance with the agreed standard. But
 now, as a pip developer, being held responsible for the resulting user
 pain, and being expected to defend it, does not make me happy.
 
 I think the pain is being overrepresented and the positives are being ignored.
 The problem is the benefits of this PEP are much like the benefits of TLS
 too. For the vast majority of people they don’t notice anything different 
 except
 installing things is faster and more reliable. They don’t attribute that to 
 the
 PEP or this decision, they just internalize it as the new norm. However the
 people who this does affect will seek out why it broke and raise an issue
 citing that thing specifically. This creates a perception of lots of pain for 
 no
 gain when the reality is not that.

Donald: I don't think anyone is arguing that hosting packages on
PyPI is a bad thing and PyPI as a service has gotten a lot better
than it was a few years ago.

However, I find it troubling that we as Python developers are *forcing*
the whole Python world to put their code into PyPI.

There are plenty good reasons not to do this, and sometimes it's
even impossible if you want to stay legal (see the PEP for details).

Accordingly, we should respect those reasons make it possible for
Python packages to live elsewhere, without having our tools put
those packages into a bad light or making it harder for Python
users to install such packages than needed.

With the checksum uploaded to PyPI, the only argument against
fetching packages from other places on the Internet is reliability,
not security.

PyPI is not the only reliable file hosting system on the Internet,
so this argument is rather weak.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 09 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

2014-05-09 Thread M.-A. Lemburg

On 09.05.2014 13:44, Donald Stufft wrote:
 
 On May 9, 2014, at 4:12 AM, M.-A. Lemburg m...@egenix.com wrote:
 Donald: I don't think anyone is arguing that hosting packages on
 PyPI is a bad thing and PyPI as a service has gotten a lot better
 than it was a few years ago.
 
 Didn’t mean to imply that I thought it was otherwise.
 

 However, I find it troubling that we as Python developers are *forcing*
 the whole Python world to put their code into PyPI.
 
 Forcing is a strong word. As of right now we’re forcing you to put it onto
 PyPI if you want to install it without *any* flags to pip. 

Which is what most people expect to be able to do.

 You're more then
 capable of hosting it externally if you wish, and pip will even tell the 
 people
 who try to install it what they need to enable in order to install it. It even
 allows people to say I don't care about any of this crap, just make all the
 external stuff work.
 
 Even if pip removed the external link handling all together and required you
 to do something like:
 
 $ pip install --extra-index-url https://example.com/our-packages/ foo
 $ pip install --find-links https://example.com/our-packages/ foo
 
 We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
 discouraging people from not hosting on PyPI and providing incentives to doing
 that.

This is all true, but in reality, you are making externally hosted
packages second class citizens in Python land by requiring
extra options even for package listings that are perfectly safe
to download from other servers.

As mentioned before: I can understand that you want to make downloads
safe for users, but if a package is hosted on some other reliable
servers and pip can check that it's a valid package, there's little
to argue for not allowing such downloads.

 There are plenty good reasons not to do this, and sometimes it's
 even impossible if you want to stay legal (see the PEP for details).
 
 I re-read the reasons, I'm not sure I really agree with most (or all?) of them
 however if people really want to do it, then there is nothing stopping them.
 It's their choice and people on the *other* side of that who are installing
 these packages also get to make a choice if they want to allow it or not.

You don't have to agree with those reasons. Fact is, they exist,
prevent package authors from uploading to PyPI and we as
Python developers should respect those reasons.

With the new infrastructure, it's far more attractive to
host packages on PyPI than it was before, so people who do
not host on PyPI will have carefully thought about this.

 Accordingly, we should respect those reasons make it possible for
 Python packages to live elsewhere, without having our tools put
 those packages into a bad light or making it harder for Python
 users to install such packages than needed.
 
 I'm not sure what put it into a bad light is supposed to mean here. Is it
 about the warning? I've already removed that completely.

It's using the same unfortunate strategy that setuptools used by
requiring an option called --single-version-externally-managed
to be able to install a package without all the .pth and egg
logic (an option that pip now uses per default, BTW ;-)).

I snipped the rest of the discussion and reliability, using
unmaintained packages and projects using their own mirrors (which
should really be the standard, not an exceptional case),
because it's not really leading anywhere:

At the time we discussed the PEP, security was the main concern,
not reliability.

Now that there is a secure way to reference external distribution
files, I think we should revisit the defaults decision in the
light of the new possibilities.


BTW: The PEP mentions re-hosting tools to upload their packages
to PyPI and also says This re-hosting tool MUST be available
before automated hosting-mode changes are announced to package
maintainers.  I am not aware of such tools.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 09 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

2014-05-09 Thread M.-A. Lemburg

On 09.05.2014 17:39, Donald Stufft wrote:
 
 On May 9, 2014, at 9:58 AM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 09.05.2014 13:44, Donald Stufft wrote:

 On May 9, 2014, at 4:12 AM, M.-A. Lemburg m...@egenix.com wrote:
 Donald: I don't think anyone is arguing that hosting packages on
 PyPI is a bad thing and PyPI as a service has gotten a lot better
 than it was a few years ago.

 Didn’t mean to imply that I thought it was otherwise.


 However, I find it troubling that we as Python developers are *forcing*
 the whole Python world to put their code into PyPI.

 Forcing is a strong word. As of right now we’re forcing you to put it onto
 PyPI if you want to install it without *any* flags to pip. 

 Which is what most people expect to be able to do.
 
 Sure, but sometimes it’s better to make an informed choice about things being
 installed instead of having it be installed by default and sometimes it’s 
 better
 to disallow doing something at all.
 
 Further most people don’t expect an install to touch any server host other
 than PyPI. 

For some idea of most people that's probably true.

I'd argue that most people probably don't really care all that
much where their packages are coming from as long as they
are getting the packages registered with PyPI, the Python
package index, and can be sure that no one has tampered with
them.

But both arguments don't really count much, since it's all
speculation.

 As far as I’m aware none of the other package repositories
 feature this, and even with the fact we support it barely anyone even cares to
 utilize it.

Most Linux repos comes with a list of standard repos to include.

In Python land, Plone uses zc.buildout with a similar
default list of repos.

 You're more then
 capable of hosting it externally if you wish, and pip will even tell the 
 people
 who try to install it what they need to enable in order to install it. It 
 even
 allows people to say I don't care about any of this crap, just make all the
 external stuff work.

 Even if pip removed the external link handling all together and required you
 to do something like:

$ pip install --extra-index-url https://example.com/our-packages/ foo
$ pip install --find-links https://example.com/our-packages/ foo

 We still wouldn't be forcing anyone to upload things to PyPI. We are, 
 however,
 discouraging people from not hosting on PyPI and providing incentives to 
 doing
 that.

 This is all true, but in reality, you are making externally hosted
 packages second class citizens in Python land by requiring
 extra options even for package listings that are perfectly safe
 to download from other servers.

 As mentioned before: I can understand that you want to make downloads
 safe for users, but if a package is hosted on some other reliable
 servers and pip can check that it's a valid package, there's little
 to argue for not allowing such downloads.
 
 Except the fact that the only people I’ve ever seen *happy* that people can
 host packages externally are a handful (5) of people (tbh primarily you and
 Stefan) and the feedback I get from every other person around the web
 in unequivocally yes please get rid of externally hosted files, they’ve done
 nothing but cause me pain.
 
 It’s not reasonable to allow a minority of users to negatively impact the 
 majority.

I think you're getting this wrong:

There *are* package authors who would like to host PyPI indexed
packages external to PyPI and they all have good reasons to
do so. It may well be a minority, but that's fine.

Changing the default to allow secure external downloads is *not*
impacting any majority.

At worst, it's impacting the users of those packages, but that's
really within the realm and responsibility of the package authors,
not PyPI or the infrastructure team, so I don't understand why you
are strongly objecting this.

 There are plenty good reasons not to do this, and sometimes it's
 even impossible if you want to stay legal (see the PEP for details).

 I re-read the reasons, I'm not sure I really agree with most (or all?) of 
 them
 however if people really want to do it, then there is nothing stopping them.
 It's their choice and people on the *other* side of that who are installing
 these packages also get to make a choice if they want to allow it or not.

 You don't have to agree with those reasons. Fact is, they exist,
 prevent package authors from uploading to PyPI and we as
 Python developers should respect those reasons.
 
 No I disagree that they are reasons at all. Half of the ones you enumerated 
 are
 nonsensical or irrelevant, the other half can be fixed by adding features to 
 or
 fixing PyPI. One or two read like reasons why someone might actually decide
 not to upload something to PyPI but which that reason isn’t really all that
 reasonable and finally a grand total of one or two of them look like an 
 actual legit
 reasons and it only applies to Crypto software.

I guess continuing this discussion doesn't really make any

Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

Well, to be fair and leaving aside uptime concerns and the general
desire to always install packages from some server instead of
a safe and trusted local directory (probably too obvious ;-),
it would certainly be possible to add support for
trusted externally hosted packages.

However, for some reason there's a strong resistance against
doing this, which I frankly don't understand.

I agree with Stefan that the warning message wording is less
than ideal. You'd normally call such blanket statements FUD,
esp. since there are plenty external hosting services which
are reliable and safe to use.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

On 08.05.2014 15:58, Donald Stufft wrote:

On May 8, 2014, at 9:39 AM, M.-A. Lemburg m...@egenix.com wrote:

Well, to be fair and leaving aside uptime concerns and the general
desire to always install packages from some server instead of
a safe and trusted local directory (probably too obvious ;-),
it would certainly be possible to add support for
trusted externally hosted packages.

There is support for trusted externally hosted packages, you put the URL in
PyPI and include a hash in the fragment like so:

http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56

The hash can be md5 or any of the sha-2 family.

Now this does not mean that ``pip install cdecimal`` will automatically
install
this, because whether or not you're willing to install from servers other than
PyPI[1] is a policy decision for the end user of pip.

Hmm, if you call that feature trusted externally hosted packages,
pip should really do trust them, right ? ;-)

I can understand that pip defaults to not trusting URLs which don't
meet the above feature requirements, but not that it still warns
about unreliable externally hosted packages even if the above
feature is used.

At the moment, pip will refuse to use an externally hosted files even
if the package author uses the above hashed URLs; even with HTTPS
and proper SSL certificate chain.

The only real contention
point there is whether installing from other servers should be on or off by
default. PEP438 selected off by default, and I agree with that decision.
Installing externally hosted files, which are able to be safely downloaded[2],
was a surprising behavior to *everyone* I've talked to who hadn't already
discovered that pip/easy_install did that. For the people it wasn't surprising
too, they said it was surprising when they had originally discovered it[3].

[1] To be specific, other than the configured index(es), which happens to
default to PyPI.
[2] For the definition of safe that PyPI/pip operate under, which is that the
author of a package is assumed to be trusted by the person electing to
download their package.
[3] I suspect people who were around when PyPI *couldn't* host files and were
only an index would be the exception to this.

However, for some reason there's a strong resistance against
doing this, which I frankly don't understand.

I agree with Stefan that the warning message wording is less
than ideal. You'd normally call such blanket statements FUD,
esp. since there are plenty external hosting services which
are reliable and safe to use.

I don't think the warning is FUD, and it doesn't mention anything security
related at all. The exact text of the warning is in the subject of the email
here:

cdecimal an externally hosted file and may be unreliable

Which is true as far as I can tell, it is externally hosted, and it may be
unreliable[1]. If there is a better wording for that I’m happy to have it and
will gladly commit it myself to pip.

The current version of pip writes:

Downloading/unpacking pkg
Could not find any downloads that satisfy the requirement pkg
Some externally hosted files were ignored (use --allow-external pkg to allow).
Cleaning up...
No distributions at all found for pkg

This wording if fine, IMO. The wording Stefan quoted gets generated
for dependencies. This should probably be changed to the same wording
(including the reference to the right command line option to use).

[1] In my experience dealing with complaints of pip's users, one of their big
ones was that some dependency they use was, typically unknown to them,
hosted externally and they found out it was hosted externally because the
server it was hosted on went down.

I think that's a general problem, not one of some server being down:
users put too much trust into the dependencies of packages they use.

Regardless of how safe/reliable we make things w/r to file hosting,
this problem does not go away. It's just too easy for people to
get tricked into trusting packages they don't even know about.

Nothing we'll ever change, though. People are lazy and easily
drop all such concerns for ease of use :-(

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, May 08 2014)
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

: Try our mxODBC.Connect Python Database Interface for free ! ::

Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

On 08.05.2014 15:57, Nick Coghlan wrote:
 On 8 May 2014 23:39, M.-A. Lemburg m...@egenix.com wrote:
 However, for some reason there's a strong resistance against
 doing this, which I frankly don't understand.
 
 Because we're taking responsibility for the end-to-end user experience
 of PyPI, and are expressly trying to eliminate the elements of that
 user experience that are beyond the control of the PyPI admin team

Oh, I guess you'd have to rewrite most of those 40k packages then :-)

Seriously, the word eliminate in there does not sit well with
our goals for openness. External services like github, sourceforge,
bitbucket, dropbox, cdns, etc. are not per-se evil and unreliable.

pip should acknowledge this and not try to eliminate all hosting
services in the world per default [sound of Empire Strikes
Back theme] ;-)

 (even the question of does this software actually work? is in our
 sights if you consider a long enough time span). That's hard enough
 with just a couple of service providers (Fastly and Rackspace) in the
 mix - it quickly becomes impossible if every new dependency from an
 installation request adds a new point of failure.

Like I said: the best option is to use a local directory which
only contains packages files that you have inspected and
actually trust :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

On 08.05.2014 16:42, M.-A. Lemburg wrote:
On 08.05.2014 15:58, Donald Stufft wrote:

On May 8, 2014, at 9:39 AM, M.-A. Lemburg m...@egenix.com wrote:

There is support for trusted externally hosted packages, you put the URL in
PyPI and include a hash in the fragment like so:

http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56

The hash can be md5 or any of the sha-2 family.

Now this does not mean that ``pip install cdecimal`` will automatically
install
this, because whether or not you're willing to install from servers other
than
PyPI[1] is a policy decision for the end user of pip.

Hmm, if you call that feature trusted externally hosted packages,
pip should really do trust them, right ? ;-)

At the moment, pip will refuse to use an externally hosted files even
if the package author uses the above hashed URLs; even with HTTPS
and proper SSL certificate chain.

Could this perhaps be changed/reconsidered for pip ?

Note that easy_install/setuptools does not have such problems.

--
Marc-Andre Lemburg
eGenix.com

: Try our mxODBC.Connect Python Database Interface for free ! ::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-20 Thread M.-A. Lemburg

On 18.04.2014 23:03, Ezio Melotti wrote:
 Hi,
 
 On Thu, Apr 17, 2014 at 9:09 PM, Brett Cannon bcan...@gmail.com wrote:


 On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić
 jurko.gospodne...@pke.hr wrote:

Hi.

 On 14.4.2014. 23:51, Brett Cannon wrote:
 Now the question is whether the maintenance cost of having to rebuild
 Python for a select number of stdlib modules is enough to warrant
 putting in the effort to make this work.

I would really love to have better startup times in production, but I
 would also really hate to lose the ability to hack around in stdlib
 sources during development just to get better startup performance.

In general, what I really like about using Python for software
 development is the ability to open any stdlib file and easily go poking
 around using stuff like 'import pdb;pdb.set_trace()' or simple print
 statements. Researching mysterious behaviour is generally much much
 MUCH! easier (read: takes less hours/days/weeks) if it ends up leading
 into a stdlib Python module than if it takes you down into the bowels of
 some C module (think zipimport.c *grin*). Not to mention the effect that
 being able to quickly resolve a mystery by hacking on some Python
 internals leaves you feeling very satisfied, while having to entrench
 yourself in those internals for a long time just to find out you've made
 something foolish on your end leaves you feeling exhausted at best.


 Freezing modules does not affect the ability to use gdb. And as long as you
 set the appropriate __file__ values then tracebacks will contain even the
 file line and location.

 
 Will the tracebacks only contain the line number or the line of code too?
 
 I've seen tracebacks involving importlib,_bootstrap that didn't
 include the code, and I'm wondering if we will get something similar
 for all the other modules you are freezing:
 
 Traceback (most recent call last):
   File /tmp/foo.py, line 7, in module
 import email
   File frozen importlib._bootstrap, line 1561, in _find_and_load
   File frozen importlib._bootstrap, line 1519, in _find_and_load_unlocked
   File frozen importlib._bootstrap, line 1473, in _find_module
   File frozen importlib._bootstrap, line 1308, in find_module
   File frozen importlib._bootstrap, line 1284, in _get_loader
   File frozen importlib._bootstrap, line 1273, in _path_importer_cache
   File frozen importlib._bootstrap, line 1254, in _path_hooks
 TypeError: 'NoneType' object is not iterable

Yes, this is what you get for frozen modules.

If you'd like to play around with a frozen stdlib this, you can
have a look at PyRun (http://pyrun.org), which does this for Python 2
and will hopefully work for Python 3.4 soonish too.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread M.-A. Lemburg

On 15.04.2014 09:45, Chris Angelico wrote:
 On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon bcan...@gmail.com wrote:
 In my work environment (Python 2.7.2, all the heavy lifting done in
 C++), startup costs are dominated by dynamic linking of all our C++
 libraries and their Boost wrappers:


 Sure, but not everyone uses Boost or has long running processes where
 startup time is minuscule compared to the total execution time.

 
 Specific use-case that I can see: Mercurial. In a git vs hg shoot-out,
 git will usually win on performance, and hg is using Py2; migrating hg
 to Py3 will (if I understand the above figures correctly) widen that
 gap, so any improvement done to startup performance will give a very
 real advantage.

You might want to have a look at this project:

http://pyrun.org/

It's currently Python 2 only, but we'll try to get it to work
with Python 3.4 as well, now that freeze.py and some other bits
have been fixed to make it work again.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows installers and OpenSSL

2014-04-10 Thread M.-A. Lemburg

On 11.04.2014 03:15, Nick Coghlan wrote:
 On 10 Apr 2014 18:55, Benjamin Peterson benja...@python.org wrote:

 On Thu, Apr 10, 2014, at 14:50, Gregory P. Smith wrote:
 Yep. All binary Python distributions that bundle SSL support need
 updating.
 But... what MRAB said.

 We also *likely* have SSL certificates and SSH host keys on
 python.orginfrastructure that need to be revoked and new certs
 reissued
 *after* all of those machines have been patched and their services
 restarted. Including hg.python.org.

 SSH isn't affected, though, right?
 
 Just SSL.

And then only machines using OpenSSL 1.0.1. Not the earlier versions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Negative timedelta strings

2014-04-02 Thread M.-A. Lemburg

n 31.03.2014 21:09, Chris Barker wrote:
 On Fri, Mar 28, 2014 at 2:52 PM, Fred Drake f...@fdrake.net wrote:
 
 On Fri, Mar 28, 2014 at 5:19 PM, Greg Ewing greg.ew...@canterbury.ac.nz
 wrote:
 ISO 8601 doesn't seem to define a representation for
 negative durations, though, so it wouldn't solve the
 original problem.

 Aside from the horribleness of the ISO 8601 notation for a duration, it's
 best not to confuse the notions of duration and delta.  Notionally, a delta
 contains more information than a duration.
 
 
 and less -- really it's different.
 
 A duration would be really useful actually, for things like next month,
 etc,. IIRC, mxDateTime has something for this, but it's NOT the same as a
 timedelta.

mxDateTime has DateTimeDelta objects which represent a time delta
(in the mathematical sense) between two points in DateTime and
RelativeDateTime objects which allow defining deltas in terms
of qualifiers which are applied to the left hand side of an operation.

With RelativeDateTime you can do things like e.g.

first_of_next_month = now() + RelativeDateTime(months=+1, day=1)

There are some other concepts you can emulate with these, like e.g.
a specific time frame (DateTime + one of the above deltas), a
reoccurring time (start_time + one of the deltas + number
occurrences + exceptions), an age concept (difference between two
DateTime values expressed in RelativeDateTime terms), etc.

Some examples:

 from mx.DateTime import *

 print RelativeDateTime(months=+1, day=1)
-(+01)-01 HH:MM:SS

 print now() + RelativeDateTime(months=+1, day=1)
2014-05-01 14:49:05.83

 print Age(now(), Date(1969,4,6))
(+0044)-(+11)-(+27) (+14):(+49):(+02)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 02 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-04-09: PyCon 2014, Montreal, Canada ...7 days to go
2014-04-29: Python Meeting Duesseldorf ... 27 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Negative timedelta strings

2014-04-02 Thread M.-A. Lemburg

On 02.04.2014 15:04, Skip Montanaro wrote:
 On Wed, Apr 2, 2014 at 7:52 AM, M.-A. Lemburg m...@egenix.com wrote:
 print now() + RelativeDateTime(months=+1, day=1)
 2014-05-01 14:49:05.83
 
 I find this sort date arithmetic unintuitive, though I'm at a loss to
 come up with better logic than you have:
 
 d = Date(2014, 2, 28)
 d + RelativeDateTime(months=+1)
 mx.DateTime.DateTime object for '2014-03-28 00:00:00.00' at 1eda8c8
 d = Date(2014, 1, 31)
 d + RelativeDateTime(months=+1)
 mx.DateTime.DateTime object for '2014-03-03 00:00:00.00' at 1eda870
 
 I guess the assumption is that one month is the length in days of the
 current month, though, you wind up with situations where shorter
 months can be skipped altogether. Is there a way to talk in terms of
 months but not have short months get skipped?

I'm not really satisfied with this solution either.

The problem is that the qualifier + 1 month is not defined for
dates with don't have a corresponding day in the following month.

This should probably either raise an exception or use some
parameter to solve the problem, by e.g. choosing the last day
of the month or using the current scheme:

mxDateTime skips to the first of the next month and then advances
the number of days defined in the left hand DateTime value,
i.e. Date(2014, 2, 1) + 30 * oneDay.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 02 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-04-09: PyCon 2014, Montreal, Canada ...7 days to go
2014-04-29: Python Meeting Duesseldorf ... 27 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7

2014-03-24 Thread M.-A. Lemburg

On 23.03.2014 08:07, Nick Coghlan wrote:
 Open Questions
 ==
 
 * What are the risks associated with allowing OpenSSL to be updated to
   new feature versions in the Windows and Mac OS X binary installers for
   maintenance releases? Currently we just upgrade to the appropriate
   OpenSSL maintenance releases, rather than switching to the latest
   feature release. In particular, is it possible Windows C extensions may
   be linking against the Python provided OpenSSL module?

Python's _ssl/_hashlib modules link statically against OpenSSL in
Python 2.7, so the OpenSSL DLLs are not exposed to other extensions.

The OpenSSL version used for 2.7.6 is 0.9.8y.

Upgrading to 1.0.0 or 1.0.1 will likely need a few minor tweaks, but
not cause general breakage - at least that's my experience with
the egenix-pyopenssl distribution.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 24 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-03-29: PythonCamp 2014, Cologne, Germany ...   5 days to go
2014-04-09: PyCon 2014, Montreal, Canada ...   16 days to go
2014-04-29: Python Meeting Duesseldorf ... 36 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7

2014-03-24 Thread M.-A. Lemburg

On 24.03.2014 13:33, Antoine Pitrou wrote:
 Le 24/03/2014 10:10, M.-A. Lemburg a écrit :
 On 23.03.2014 08:07, Nick Coghlan wrote:
 Open Questions
 ==

 * What are the risks associated with allowing OpenSSL to be updated to
new feature versions in the Windows and Mac OS X binary installers for
maintenance releases? Currently we just upgrade to the appropriate
OpenSSL maintenance releases, rather than switching to the latest
feature release. In particular, is it possible Windows C extensions may
be linking against the Python provided OpenSSL module?

 Python's _ssl/_hashlib modules link statically against OpenSSL in
 Python 2.7, so the OpenSSL DLLs are not exposed to other extensions.
 
 I suppose you mean under Windows. 

Yes. Should have included that detail in the email :-)

 Under Linux (and probably OS X too), the _ssl module is linked
 dynamically with OpenSSL:
 
 $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
 linux-vdso.so.1 =  (0x7fff3f1de000)
 libssl.so.1.0.0 = /lib/x86_64-linux-gnu/libssl.so.1.0.0 
 (0x7fd8853ea000)
 libcrypto.so.1.0.0 = /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 
 (0x7fd88501)
 libpthread.so.0 = /lib/x86_64-linux-gnu/libpthread.so.0 
 (0x7fd884df1000)
 libc.so.6 = /lib/x86_64-linux-gnu/libc.so.6 (0x7fd884a2b000)
 libdl.so.2 = /lib/x86_64-linux-gnu/libdl.so.2 (0x7fd884827000)
 /lib64/ld-linux-x86-64.so.2 (0x7fd885868000)

Right, and it's using the system library, not a private copy - which
can be both good and bad depending on how recent the system's library
version is.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 24 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-03-29: PythonCamp 2014, Cologne, Germany ...   5 days to go
2014-04-09: PyCon 2014, Montreal, Canada ...   16 days to go
2014-04-29: Python Meeting Duesseldorf ... 36 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7

2014-03-24 Thread M.-A. Lemburg

On 24.03.2014 18:23, Ned Deily wrote:
 In article 
 cadisq7f0cnzrfm4i8xj13j+slq63uynqkdo12czm5yeq3bf...@mail.gmail.com,
  Nick Coghlan ncogh...@gmail.com wrote:
 You also reminded me that I need to dig around for and reference Ned's
 email about the status of OS X and reference that (OpenSSL upgrades
 were a casualty of Apple's anti-GPL crusade, so the OS X installers
 were switched to static linking somewhere along the line).
 
 AFAIK, Apple's decision to deprecate OpenSSL has nothing to do with the 
 GPL, since OpenSSL isn't GPL-licensed, but rather with OpenSSL API 
 compatibility issues:
 
 http://rentzsch.tumblr.com/post/33696323211/wherein-i-write-apples-techno
 te-about-openssl-on-os-x

What a strange reasoning. Do they really believe that ABIs don't
change when bumping the library version from 0.9.8 to 1.0.0 ?

OpenSSL's history w/r to backwards compatibility up until 0.9.7 wasn't
the greatest, but since 0.9.8 it has gotten to a level that's very
reliable.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 24 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-03-29: PythonCamp 2014, Cologne, Germany ...   5 days to go
2014-04-09: PyCon 2014, Montreal, Canada ...   16 days to go
2014-04-29: Python Meeting Duesseldorf ... 36 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

2014-03-22 Thread M.-A. Lemburg

On 22.03.2014 22:11, Nick Coghlan wrote:
 PEP: 466
 Title: Network Security Enhancement Exception for All Branches

+1

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 22 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-03-29: PythonCamp 2014, Cologne, Germany ...   7 days to go
2014-04-09: PyCon 2014, Montreal, Canada ...   18 days to go
2014-04-29: Python Meeting Duesseldorf ... 38 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

2014-02-18 Thread M.-A. Lemburg

On 18.02.2014 05:25, Tim Peters wrote:
 [M.-A. Lemburg]
 Now, the choice to have None compare less than all other objects
 may have been arbitrary, but IMO it was a good, consistent and
 useful choice.
 
 Possibly useful for some apps, sure.  Not for my apps.  For example,
 when I initialize an object attribute to None in Python 3, I _expect_
 I'll get an exception if I try to use that attribute in most contexts.
  It makes no more sense to ask whether that attribute is, say, less
 than 3, than it does to add 3 to it.  The exception is most useful
 then.  More often than not, it was annoying to me that Python 2
 _didn't_ whine about trying to compare None.

Yes, I see your point, but please also consider that None is
used in database applications as natural mapping for SQL NULL
and in other data processing applications to mean no value.

This is not garbage data, it's just missing information for
a particular field and you can still happily process such data
without having the information for these fields as values
(indeed in some cases, the information whether a field has a
value or not is important, e.g. for data validations and to cross
check entries).

You do still want to be able to sort such data, but as it stands,
Python 3 doesn't allow you to, without adding an extra layer
of protection against None values deep inside the structures
you're sorting.

 So why not bring it back
 
 A huge barrier is (or should be) that Python 3 is over 5 years old
 now.  Fiddling with the semantics of basic builtin types was possible
 - and even welcome - 6 years ago.  Now they really shouldn't be
 touched in the absence of a critical bug, or a wholly
 backward-compatible (with Python 3) addition.
 
 and perhaps this time in a way that actually does work consistently for
 all Python objects by implementing the tp_richcompare slot on
 PyNoneType objects and documenting it ?!
 
 Something to suggest for Python 4, in which case I'll only be -0 ;-)

Well, like I said: we'd be making something work again that has
worked before and only remove a failure case. The barrier for
entry should be lower in such a case.

This is similar to bringing back the u literals. Those used
to cause a SyntaxError and now they no longer do.

Even better: we have a chance to properly document the behavior
this time around and make it consistent all the way.

The alternative would be adding a new singleton to mean mostly
the same thing as None, but having the property of comparing
less than all other objects and then recommend its use in the
DB-API for Python 3 applications... which would break a lot
of existing DB-API based apps of course... which is the reason
why I'm advocating for changing None instead :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 18 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 15.02.2014 07:03, Stephen J. Turnbull wrote:
 M.-A. Lemburg writes:
 
   IMO, it was a mistake to have None return a TypeError in
   comparisons, since it makes many typical data operations
   fail, e.g.
 
 I don't understand this statement.  The theory is that they *should*
 fail.
 
 The example of sort is a good one.  Sometimes you want missing values
 to be collected at the beginning of a list, sometimes at the end.
 Sometimes you want them treated as top elements, sometimes as bottom.
 And sometimes it is a real error for missing values to be present.
 Not to mention that sometimes the programmer simply hasn't thought
 about the appropriate policy.  I don't think Python should silently
 impose a policy in that case, especially given that the programmer may
 have experience with any of the above treatments in other contexts.

None is special in Python and has always (and intentionally) sorted
before any other object. In data processing and elsewhere in Python
programming, it's used to signal: no value available.

Python 3 breaks this notion by always raising an exception when
using None in an ordered comparison, making it pretty much useless
for the above purpose.

Yes, there are ways around this, but none of them are intuitive.

Here's a particularly nasty case:

 l = [(1, None), (2, None)]
 l.sort()
 l
[(1, None), (2, None)]

 l = [(1, None), (2, None), (3, 4)]
 l.sort()
 l
[(1, None), (2, None), (3, 4)]

 l = [(1, None), (2, None), (3, 4), (2, 3)]
 l.sort()
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: unorderable types: int()  NoneType()

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 17.02.2014 12:23, Gustavo Carneiro wrote:
 On 17 February 2014 11:14, M.-A. Lemburg m...@egenix.com wrote:
 
 On 15.02.2014 07:03, Stephen J. Turnbull wrote:
 M.-A. Lemburg writes:

   IMO, it was a mistake to have None return a TypeError in
   comparisons, since it makes many typical data operations
   fail, e.g.

 I don't understand this statement.  The theory is that they *should*
 fail.

 The example of sort is a good one.  Sometimes you want missing values
 to be collected at the beginning of a list, sometimes at the end.
 Sometimes you want them treated as top elements, sometimes as bottom.
 And sometimes it is a real error for missing values to be present.
 Not to mention that sometimes the programmer simply hasn't thought
 about the appropriate policy.  I don't think Python should silently
 impose a policy in that case, especially given that the programmer may
 have experience with any of the above treatments in other contexts.

 None is special in Python and has always (and intentionally) sorted
 before any other object. In data processing and elsewhere in Python
 programming, it's used to signal: no value available.

 Python 3 breaks this notion by always raising an exception when
 using None in an ordered comparison, making it pretty much useless
 for the above purpose.

 Yes, there are ways around this, but none of them are intuitive.

 Here's a particularly nasty case:

 l = [(1, None), (2, None)]
 l.sort()
 l
 [(1, None), (2, None)]

 l = [(1, None), (2, None), (3, 4)]
 l.sort()
 l
 [(1, None), (2, None), (3, 4)]

 l = [(1, None), (2, None), (3, 4), (2, 3)]
 l.sort()
 Traceback (most recent call last):
   File stdin, line 1, in module
 TypeError: unorderable types: int()  NoneType()


 Maybe Python 3 should have a couple of None-like objects that compare the
 way you want: AlwaysComparesLess and AlwaysComparesGreater, but with better
 names (maybe 'PlusInfinity' and 'MinusInfinity'?).  Just leave None alone,
 please.

This doesn't only apply to numeric comparisons. In Python 2 you
can compare None with any kind of object and it always sorts first,
based on the intuition that nothing is less than anything :-)

FWIW, I don't think we need to invent a new name for it, just add
an appropriate tp_richcompare slot to the PyNoneType or readd the
special case to Object/object.c. This would also aid in porting
existing Python 2 code to Python 3.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 17.02.2014 12:47, Serhiy Storchaka wrote:
 17.02.14 13:14, M.-A. Lemburg написав(ла):
 Here's a particularly nasty case:

 l = [(1, None), (2, None)]
 l.sort()
 l
 [(1, None), (2, None)]

 l = [(1, None), (2, None), (3, 4)]
 l.sort()
 l
 [(1, None), (2, None), (3, 4)]

 l = [(1, None), (2, None), (3, 4), (2, 3)]
 l.sort()
 Traceback (most recent call last):
File stdin, line 1, in module
 TypeError: unorderable types: int()  NoneType()
 
 If you replace None to another value that cannot be compared with int (e.g. 
 string), you will got
 the same nasty case.

Yes, but that's not the point. Unlike strings or other mixed types that
you cannot compare, None is used as placeholder in data processing as
special value to mean no value available.

You intentionally use such values in programming. It's not a bug to
have None in a data list or as value of a variable.

-- 
Marc-Andre Lemburg
Director
Python Software Foundation
http://www.python.org/psf/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 17.02.2014 12:49, Gustavo Carneiro wrote:
 On 17 February 2014 11:43, M.-A. Lemburg m...@egenix.com wrote:
 
 On 17.02.2014 12:23, Gustavo Carneiro wrote:
 On 17 February 2014 11:14, M.-A. Lemburg m...@egenix.com wrote:

 On 15.02.2014 07:03, Stephen J. Turnbull wrote:
 M.-A. Lemburg writes:

   IMO, it was a mistake to have None return a TypeError in
   comparisons, since it makes many typical data operations
   fail, e.g.

 I don't understand this statement.  The theory is that they *should*
 fail.

 The example of sort is a good one.  Sometimes you want missing values
 to be collected at the beginning of a list, sometimes at the end.
 Sometimes you want them treated as top elements, sometimes as bottom.
 And sometimes it is a real error for missing values to be present.
 Not to mention that sometimes the programmer simply hasn't thought
 about the appropriate policy.  I don't think Python should silently
 impose a policy in that case, especially given that the programmer may
 have experience with any of the above treatments in other contexts.

 None is special in Python and has always (and intentionally) sorted
 before any other object. In data processing and elsewhere in Python
 programming, it's used to signal: no value available.

 Python 3 breaks this notion by always raising an exception when
 using None in an ordered comparison, making it pretty much useless
 for the above purpose.

 Yes, there are ways around this, but none of them are intuitive.

 Here's a particularly nasty case:

 l = [(1, None), (2, None)]
 l.sort()
 l
 [(1, None), (2, None)]

 l = [(1, None), (2, None), (3, 4)]
 l.sort()
 l
 [(1, None), (2, None), (3, 4)]

 l = [(1, None), (2, None), (3, 4), (2, 3)]
 l.sort()
 Traceback (most recent call last):
   File stdin, line 1, in module
 TypeError: unorderable types: int()  NoneType()


 Maybe Python 3 should have a couple of None-like objects that compare the
 way you want: AlwaysComparesLess and AlwaysComparesGreater, but with
 better
 names (maybe 'PlusInfinity' and 'MinusInfinity'?).  Just leave None
 alone,
 please.

 This doesn't only apply to numeric comparisons. In Python 2 you
 can compare None with any kind of object and it always sorts first,
 based on the intuition that nothing is less than anything :-)

 I still think that relying on your intuition is not the right way for
 Python.  Refuse the temptation to guess.

Hmm, are you sure ?

There should be one-- and preferably only one --obvious way to do it.

and then

Although that way may not be obvious at first unless you're Dutch.

which directly relates to:

changeset:   16123:f997ded4e219
branch:  legacy-trunk
user:Guido van Rossum gu...@python.org
date:Mon Jan 22 19:28:09 2001 +
summary: New special case in comparisons: None is smaller than any other 
object

:-)

 FWIW, I don't think we need to invent a new name for it, just add
 an appropriate tp_richcompare slot to the PyNoneType or readd the
 special case to Object/object.c. This would also aid in porting
 existing Python 2 code to Python 3.
 
 Based on your comment, SortsFirst and SortsLast sound like good names ;-)
 
 These would be universal sortable objects, that could be compared to any
 other type.

Of course, it's easy to add a new type for this, but a lot of Python 2
code relies on None behaving this way, esp. code that reads data from
databases, since None is the Python mapping for SQL NULL.

-- 
Marc-Andre Lemburg
Director
Python Software Foundation
http://www.python.org/psf/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 17.02.2014 13:12, Nick Coghlan wrote:
 On 17 Feb 2014 21:15, M.-A. Lemburg m...@egenix.com wrote:

 On 15.02.2014 07:03, Stephen J. Turnbull wrote:
 M.-A. Lemburg writes:

   IMO, it was a mistake to have None return a TypeError in
   comparisons, since it makes many typical data operations
   fail, e.g.

 I don't understand this statement.  The theory is that they *should*
 fail.

 The example of sort is a good one.  Sometimes you want missing values
 to be collected at the beginning of a list, sometimes at the end.
 Sometimes you want them treated as top elements, sometimes as bottom.
 And sometimes it is a real error for missing values to be present.
 Not to mention that sometimes the programmer simply hasn't thought
 about the appropriate policy.  I don't think Python should silently
 impose a policy in that case, especially given that the programmer may
 have experience with any of the above treatments in other contexts.

 None is special in Python and has always (and intentionally) sorted
 before any other object. In data processing and elsewhere in Python
 programming, it's used to signal: no value available.
 
 This is the first I've ever heard of that sorting behaviour being an
 intentional feature, rather than just an artefact of Python 2 allowing
 arbitrarily ordered comparisons between different types. Can you point me
 to the relevant entry in the Python 2 language reference?

This is not documented anywhere in the language spec, AFAIK. It
is documented in the code (Python 2.7; Object/object.c):

default_3way_compare(PyObject *v, PyObject *w)
...
/* None is smaller than anything */
if (v == Py_None)
return -1;
if (w == Py_None)
return 1;

Note that it's not important whether None is smaller or greater
than any other object. The important aspect is that it's sorting
order is consistent and doesn't raise a TypeError when doing an
ordered comparison with other objects.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 17.02.2014 13:19, Serhiy Storchaka wrote:
 17.02.14 13:56, M.-A. Lemburg написав(ла):
 Yes, but that's not the point. Unlike strings or other mixed types that
 you cannot compare, None is used as placeholder in data processing as
 special value to mean no value available.
 
 Isn't float('nan') such placeholder?

You can easily construct other such placeholders, but None was intended
for this purpose:

http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None


The Python None object, denoting lack of value. ...


 You intentionally use such values in programming. It's not a bug to
 have None in a data list or as value of a variable.
 
 You can't have None in array('f'), you can't add or multiply by None. 
 Relation operators don't looks
 an exception here. Applying sorted() to a list which contains numbers and 
 Nones makes as much sense
 as applying sum() to it.

Of course, you cannot apply any operations with None - it doesn't
have a value -, but you can compare it to other objects and it provides
a consistent behavior in Python 2. Python 3 is missing such an object.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError



On 17.02.2014 14:29, Gustavo Carneiro wrote:
 On 17 February 2014 12:30, M.-A. Lemburg m...@egenix.com wrote:
 
 On 17.02.2014 13:19, Serhiy Storchaka wrote:
 17.02.14 13:56, M.-A. Lemburg написав(ла):
 Yes, but that's not the point. Unlike strings or other mixed types that
 you cannot compare, None is used as placeholder in data processing as
 special value to mean no value available.

 Isn't float('nan') such placeholder?

 You can easily construct other such placeholders, but None was intended
 for this purpose:

 http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None

 
 The Python None object, denoting lack of value. ...
 

 You intentionally use such values in programming. It's not a bug to
 have None in a data list or as value of a variable.

 You can't have None in array('f'), you can't add or multiply by None.
 Relation operators don't looks
 an exception here. Applying sorted() to a list which contains numbers
 and Nones makes as much sense
 as applying sum() to it.

 Of course, you cannot apply any operations with None - it doesn't
 have a value -, but you can compare it to other objects and it provides
 a consistent behavior in Python 2. Python 3 is missing such an object.

 
 I agree with you that Python 3 could use such an object.  Just don't make
 it the default.  Leave None as it is.
 
 Also I agree that my previous naming suggestions are bad.  What we need is
 only one (or two) additional object whose main semantic meaning is still
 no value, but which also adds a meaning of comparable.  Then it's a
 matter of choosing a good name for it, with lots of bikeshedding involved.
  Just lets not change None only because we're too lazy to discuss a proper
 alternative name.  Also this use case is not _that_ common, so it's ok if
 it has a slightly longer name than None.
 
 Also think of the implications of changing None at this point.  It would
 allow us to write programs that work Python = 3.5 and Python = 2.7, but
 fail mysteriously in all other versions in between.  What a mess that would
 be...

Yes, that's unfortunately true.

If you regard this as bug in the Python 3 series, we could still fix
it for 3.3 and 3.4, though. After all, we'd just be removing an unwanted
exception and not make already working Python 3 applications fail.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 17.02.2014 15:38, Jon Ribbens wrote:
 On Mon, Feb 17, 2014 at 12:43:25PM +0100, M.-A. Lemburg wrote:
 This doesn't only apply to numeric comparisons. In Python 2 you
 can compare None with any kind of object and it always sorts first,
 
 No you can't. See http://bugs.python.org/issue1673405 .
 
 According to Tim Peters, the None is less than everything rule
 never existed.

Well, then Tim probably didn't read the code in object.c :-)

Seriously, the datetime module types were the first types to experiment
with the new mixed type operation mechanism added at the time:

http://www.python.org/dev/peps/pep-0207/
http://www.python.org/dev/peps/pep-0208/

Objects implementing the rich comparison slot can indeed override
these defaults and also raise exceptions in comparison operations,
so you're right: None comparisons can still be made to raise
exceptions, even in Python 2.7.

Still, None comparisons work just fine for most types in Python 2.x
and people have written code assuming that it works for many years.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

On 17.02.2014 21:12, Tim Peters wrote:
 [...]

 Guido wanted to drop all the arbitrary but consistent mixed-type
 comparison crud for Python 3. Nothing special about None in that.  As
 already noted, the various `datetime` types were the first to
 experiment with implementing full blown Python3-ish mixed-type
 comparison rules.  After the first two times that caught actual bugs
 in code using the new types, there was no turning back.  It's not so
 much that Python 3 finished the job as that Python 2 started it ;-)

Well, I guess it depends on how you look at it.

None worked as compares less than all other objects simply due
to the fact that None is a singleton and doesn't implement the
comparison slots (which for all objects not implementing rich
comparisons, meant that the fallback code triggered in Python 2).

In Python 3 the special casing was dropped and because None
still doesn't implement the comparison slot (tp_richcompare this time),
it doesn't support ordering comparisons anymore.

Now, the choice to have None compare less than all other objects
may have been arbitrary, but IMO it was a good, consistent and
useful choice.

So why not bring it back and perhaps this time in a way that
actually does work consistently for all Python objects by
implementing the tp_richcompare slot on PyNoneType objects
and documenting it ?!

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 17 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

2014-02-14 Thread M.-A. Lemburg

On 14.02.2014 11:20, Antoine Pitrou wrote:
 On Fri, 14 Feb 2014 20:13:43 +1000
 Nick Coghlan ncogh...@gmail.com wrote:
 On 14 February 2014 20:02, Antoine Pitrou solip...@pitrou.net wrote:
 On Fri, 14 Feb 2014 10:46:50 +0100
 Lennart Regebro rege...@gmail.com wrote:

 Sending this to python-dev as I'm wondering if this was considered when 
 the
 choice to have objects of different types raise a TypeError when 
 ordered...

 So, the concrete I case I have is implementing stable ordering for the
 python Range objects that psycopg2 uses. These have 3 attributes that can
 either be None or, for sake of argument, a numeric value.

 [...]

 It was considered. It's not obvious where you want None to appear in
 your ordering, so you will have to implement this by yourself. I can't
 come up with anything obviously shorter.

 I have to agree with Lennart. The fact that SQL defines an order for
 NULL and other values doesn't mean it's obvious or right in any way (I
 never remember which way it goes).

 SQL doesn't define an order for NULL, it's more like a quiet NaN -
 if either operand is NULL, the result is also NULL. (I ran into this
 recently, in the context of NULL == value and NULL != value both
 being effectively false).
 
 Hmm, it seems you're right, but I'm quite sure some DBMSes have a
 consistent way of ordering NULLs when using ORDER BY on a nullable
 column.

They do, but it's not consistent:

MS SQL Server: orders NULLs first (in table order; stable sort)
Sybase ASE:orders NULLs first (in arbitrary order)
Oracle:orders NULLs last (in arbitrary order)
PostgreSQL:orders NULLs last (in arbitrary order)
IBM DB2:   orders NULLs last (in table order; stable sort)

Reference: tests done with actual databases.

A note about consistency: None is always ordered first in Python 2,
so there is a precedent. And since Python's list.sort() is stable,
Python 2 is in the same camp as MS SQL Server.

From Python-2.7.6/Objects/object.c:

/* None is smaller than anything */
if (v == Py_None)
return -1;
if (w == Py_None)
return 1;

IMO, it was a mistake to have None return a TypeError in
comparisons, since it makes many typical data operations
fail, e.g.

Python2:
 l = [1,2,None,4,5,None,6]
 l.sort()
 l
[None, None, 1, 2, 4, 5, 6]

Python3:
 l = [1,2,None,4,5,None,6]
 l.sort()
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: unorderable types: NoneType()  int()

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 14 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enable Hostname and Certificate Chain Validation

2014-01-22 Thread M.-A. Lemburg

On 22.01.2014 11:30, Donald Stufft wrote:
 I would like to propose that a backwards incompatible change be made to 
 Python to make
 verification of hostname and certificate chain the default instead of 
 requiring it to be opt
 in.
 
 Python 3.4 has made great strides in making it easier for applications to 
 simply turn on these
 settings, however many people are not aware at all that they need to opt into 
 this. Most assume
 that it will operate similarly to their browser, curl, wget, etc and validate 
 by default and
 in the typical style of security related issues it will appear to work just 
 fine however be
 grossly insecure.
 
 In the real world “opt in security” typically translates to just plain old 
 insecure for the
 bulk of applications/libraries. I believe that Python has a responsibility to 
 do the right
 thing by default here and it is in the best position to do so. The 
 alternative requires every
 Python developer who wants to access a secure resource to be educated on the 
 fact that they
 need to flip some switch to do what most of them would expect.

Such a change would introduce considerable breakage. This would either
have to be done using our usual pending deprecation, deprecation, removal
dance (over three releases) or be postponed until Python 4.

Note that several python.org services use CAcerts which would no
longer be accessible per default following such a change.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/ 
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/ 
 mxODBC, mxDateTime,
 mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enable Hostname and Certificate Chain Validation

2014-01-22 Thread M.-A. Lemburg

On 22.01.2014 11:56, Donald Stufft wrote:
 
 On Jan 22, 2014, at 5:51 AM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 22.01.2014 11:30, Donald Stufft wrote:
 I would like to propose that a backwards incompatible change be made to 
 Python to make
 verification of hostname and certificate chain the default instead of 
 requiring it to be opt
 in.

 Python 3.4 has made great strides in making it easier for applications to 
 simply turn on these
 settings, however many people are not aware at all that they need to opt 
 into this. Most assume
 that it will operate similarly to their browser, curl, wget, etc and 
 validate by default and
 in the typical style of security related issues it will appear to work just 
 fine however be
 grossly insecure.

 In the real world “opt in security” typically translates to just plain old 
 insecure for the
 bulk of applications/libraries. I believe that Python has a responsibility 
 to do the right
 thing by default here and it is in the best position to do so. The 
 alternative requires every
 Python developer who wants to access a secure resource to be educated on 
 the fact that they
 need to flip some switch to do what most of them would expect.

 Such a change would introduce considerable breakage. This would either
 have to be done using our usual pending deprecation, deprecation, removal
 dance (over three releases) or be postponed until Python 4.
 
 I can understand the need for doing the typical deprecation dance, although
 I believe such policies are often overlooked or accelerated for security
 sensitive changes. I do believe that waiting until Python 4 would be doing
 a great disservice to the users of Python though.

Well, it's not really a security issue, since the security features
are present in Python 3.4. It's just that the user has to enable them.

 Note that several python.org services use CAcerts which would no
 longer be accessible per default following such a change.
 
 Not that it has much to do with this proposal, but those services should
 be switched to use certificates that are well trusted.

Oh, it does have to do with the proposal, since it's a prominent
example of services that would no longer work. While we can potentially
change the situation for python.org servers, you have the same issues
with other services which we don't have any influence on.

The change would also disable all services using self-signed
certificates which are very common in internal networks and
for ad-hoc setups. Many routers and other devices use self-signed
certificates when offering HTTPS services.

I think overall, it's good to have default security, but locking out
all certificates which do not have their root CA certs installed
in default installations of systems per default would likely lead to
people seeking other more insecure ways of getting things to work,
rather than asking their admins to add their CA certs to the certificate
chain configuration. So I'm not sure whether raising errors is the
best way to achieve better default security. Perhaps just using
warnings would be better.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enable Hostname and Certificate Chain Validation

2014-01-22 Thread M.-A. Lemburg

On 22.01.2014 12:36, Donald Stufft wrote:
 
 On Jan 22, 2014, at 6:30 AM, M.-A. Lemburg m...@egenix.com wrote:
 The change would also disable all services using self-signed
 certificates which are very common in internal networks and
 for ad-hoc setups. Many routers and other devices use self-signed
 certificates when offering HTTPS services.
 
 It will just disable them by default, they can still easily be accessed
 you’d just need to pass the “do not verify” flag. This clearly indicates
 that you’re opting out of the S in HTTPS.
 

 I think overall, it's good to have default security, but locking out
 all certificates which do not have their root CA certs installed
 in default installations of systems per default would likely lead to
 people seeking other more insecure ways of getting things to work,
 rather than asking their admins to add their CA certs to the certificate
 chain configuration. So I'm not sure whether raising errors is the
 best way to achieve better default security. Perhaps just using
 warnings would be better.
 
 Again it’s not “locking out”, it’s simply “requires explicitly saying
 I want to not validate”.

This would have to configurable without changing application code,
e.g. using an environment settings.

Otherwise, you do lock out existing scripts and applications from
using Python 3.6 by requiring supporting custom configurations
from all of them.

Simply saying: oh, just change your code to never validate is not
a good solution either.

 It’s hard to be more insecure than not verifying. Just about the only
 other way is to use plaintext but the only real difference there is
 passive vs attack attacks. However typically if you’re in the situation
 to do a passive attack you can also do an active attack. So “more
 insecure” is minorly more insecure while an error is drastically
 more secure.

I disagree with that statement. Using HTTPS without verification
is still far more secure than using plain text.

I know that verification is a lot better, but please remember
that practicality beats purity.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

2014-01-13 Thread M.-A. Lemburg

On 13.01.2014 07:51, Nick Coghlan wrote:

 [Using a new asciistr type]

 The key thing that the text model change in Python 3 enabled is for us
 to use the type system to *help* with managing the complexity of
 dealing with text encodings. We've got a long way with just the two
 pure types, and no additional types that straddle the binary/text
 boundary the way the Python 2 str type did. Unlike introducing *new*
 ASCII-only operations to the bytes type, adding new types specifically
 for dealing with ASCII compatible formats (especially starting life as
 a third party library) isn't compromising the Python 3 text model,
 it's embracing it and making it work for us (which is why I've been
 suggesting that it be considered since at least 2010). The problem
 with str in Python 2 was that one type was used to represent too
 many things with serious semantic differences.
 
 The ongoing attempts to reintroduce that ambiguity to the core bytes
 type rather than exploring the creation of new types and then filing
 bugs for any interoperability issues those attempts uncover in the
 core types represents one of the worst cases of paradigm lock that I
 have ever seen :P

In theory this sounds nice, but in practice you often run into the issue
that whenever you pass such a str-subtype to some function that
works on str doesn't return the str-subtype as result, but instead
a new str object.

As a result, you have to keep track of which operations work
on your str-subtype alone and which convert it back to a str,
making the approach infeasible for all but the most basic
uses.

This is why we try to make the basic types as useful as possible
for everyone. It's also the main reason why subtyping 8-bit strings
and Unicode in Python 2 wasn't a popular sport :-)

Leaving aside the discussion about str and bytes, I think PEP 460
has much potential of making life easier for people dealing with binary
data: the formatting codes for the bytes format methods could
be extended to include the struct module features - with the struct
module then turning into a proxy for these new format methods (much
like we did with the string module when string methods were
introduced).


BTW: There's a little known trick in Python 2 which also lets you
disable the string to Unicode coercion: all you have to do is
set the default encoding to undefined (see site.py:setencoding()).
Python 2 will then raise a UnicodeError whenever coercion would trigger.
I added that codec to experiment with this scenario in the early days
of the Unicode integration.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 13 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5

2014-01-11 Thread M.-A. Lemburg

On 11.01.2014 14:54, Georg Brandl wrote:
 Am 11.01.2014 14:49, schrieb Georg Brandl:
 Am 11.01.2014 10:44, schrieb Stephen Hansen:

 I mean, its not like the bytes type lacks knowledge of the subset of bytes
 that happen to be 7-bit ascii-compatible and can't perform text-ish 
 operations
 on them--

   Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:18:40) [MSC v.1600 32 
 bit
 (Intel)] on win32
   Type help, copyright, credits or license for more information.
bstephen hansen.title()
   b'Stephen Hansen'

 How is this not a practical recognition that yes, while bytes are byte 
 streams
 and not text, a huge subset of bytes are text-y, and as long as we maintain 
 the
 barrier between higher characters and implicit conversion therein, we're 
 fine?

 I don't see the difference here. There is a very real, practical need to
 interpolate bytes. This very real, practical need includes the very real
 recognition that converting 12345 to b'12345' is not something weird, 
 unusual,
 and subject to the thorny issues of Encodings. It is not violating the 
 doctrine
 of separation of powers between Text and Bytes.

 This. Exactly. Thanks for putting it so nicely, Stephen.
 
 To elaborate: if the bytes type didn't have all this ASCII-aware functionality
 already, I think we would have (and be using) a dedicated asciistr type 
 right
 now.  But it has the functionality, and it's way too late to remove it.

I think we need to step back a little from the purist view
of things and give more emphasis on the practicality beats
purity Zen.

I complete agree with Stephen, that bytes are in fact often
an encoding of text. If that text is ASCII compatible, I don't
see any reason why we should not continue to expose the C lib
standard string APIs available for text manipulations on bytes.

We don't have to be pedantic about the bytes/text separation.
It doesn't help in real life.

If you give programmers the choice they will - most of the time -
do the right thing. If you don't give them the tools, they'll
work around the missing features in a gazillion different
ways of which many will probably miss a few edge cases.

bytes already have most of the 8-bit string methods from Python 2,
so it doesn't hurt adding some more of the missing features
from Python 2 on top to make life easier for people dealing
with multiple/unknown encoding data.

BTW: I don't know why so many people keep asking for use cases.
Isn't it obvious that text data without known (but ASCII compatible)
encoding or multiple different encodings in a single data chunk
is part of life ? Most HTTP packets fall into this category,
many email messages as well. And let's not forget that we don't
live in a perfect world. Broken encodings are everywhere around
you - just have a look at your spam folder for a decent chunk
of example data :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 11 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5

2014-01-11 Thread M.-A. Lemburg

On 11.01.2014 16:34, Nick Coghlan wrote:
On 12 January 2014 01:15, M.-A. Lemburg m...@egenix.com wrote:
On 11.01.2014 14:54, Georg Brandl wrote:
Am 11.01.2014 14:49, schrieb Georg Brandl:
Am 11.01.2014 10:44, schrieb Stephen Hansen:

I mean, its not like the bytes type lacks knowledge of the subset of
bytes
that happen to be 7-bit ascii-compatible and can't perform text-ish
operations
on them--

Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:18:40) [MSC v.1600
32 bit
(Intel)] on win32
Type help, copyright, credits or license for more information.
bstephen hansen.title()
b'Stephen Hansen'

How is this not a practical recognition that yes, while bytes are byte
streams
and not text, a huge subset of bytes are text-y, and as long as we
maintain the
barrier between higher characters and implicit conversion therein, we're
fine?

I don't see the difference here. There is a very real, practical need to
interpolate bytes. This very real, practical need includes the very real
recognition that converting 12345 to b'12345' is not something weird,
unusual,
and subject to the thorny issues of Encodings. It is not violating the
doctrine
of separation of powers between Text and Bytes.

This. Exactly. Thanks for putting it so nicely, Stephen.

To elaborate: if the bytes type didn't have all this ASCII-aware
functionality
already, I think we would have (and be using) a dedicated asciistr type
right
now. But it has the functionality, and it's way too late to remove it.

I think we need to step back a little from the purist view
of things and give more emphasis on the practicality beats
purity Zen.

I complete agree with Stephen, that bytes are in fact often
an encoding of text. If that text is ASCII compatible, I don't
see any reason why we should not continue to expose the C lib
standard string APIs available for text manipulations on bytes.

We don't have to be pedantic about the bytes/text separation.
It doesn't help in real life.

Yes, it bloody well does. The number of people who have told me that
using Python 3 is what allowed them to finally understand how Unicode
works vastly exceeds the number of wire protocol and file format devs
that have complained about working with binary formats being
significantly less tolerant of the it's really like ASCII text
mindset.

We are NOT going back to the confusing incoherent mess that is the
Python 2 model of bolting Unicode onto the side of POSIX:
http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html#what-actually-changed-in-the-text-model-between-python-2-and-python-3

While that was an *expedient* (and, in fact, necessary) solution at
the time, the fact it is still thoroughly confusing people 13 years
later shows it is not a *comprehensible* solution.

FWIW: I quite liked the Python 2 model, but perhaps that's because
I already knww how Unicode works, so could use it to make my
life easier ;-)

Seriously, Unicode has always caused heated discussions and
I don't expect this to change in the next 5-10 years.

The point is: there is no 100% perfect solution either way and
when you acknowledge this, things don't look black and white anymore,
but instead full of colors :-)

Python 3 forces people to actually use Unicode; in Python 2 they
could easily avoid it. It's good to educate people on how it's
used and the issues you can run into, but let's not forget
that people are trying to get work done and we all love readable
code.

PEP 460 just adds two more methods to the bytes object which come
in handy when formatting binary data; I don't think it has potential
to muddy the Python 3 text model, given that the bytes
object already exposes a dozen of other ASCII text methods :-)

If you give programmers the choice they will - most of the time -
do the right thing. If you don't give them the tools, they'll
work around the missing features in a gazillion different
ways of which many will probably miss a few edge cases.

bytes already have most of the 8-bit string methods from Python 2,
so it doesn't hurt adding some more of the missing features
from Python 2 on top to make life easier for people dealing
with multiple/unknown encoding data.

Because people that aren't happy with the current bytes type
persistently refuse to experiment with writing their own extension
type to figure out what the API should look like. Jamming speculative
API design into the core text model without experimenting in a third
party extension first is a straight up stupid idea.

Anyone that is pushing for this should be checking out Benno's first
draft experimental prototype for asciistr and be working on getting it
passing the test suite I created:
https://github.com/jeamland/asciicompat

The Wah, you broke it and now I have completely forgotten how to
create custom types, so I'm just going to piss and moan until somebody
else fixes it infantilism of the past five years

Re: [Python-Dev] Python3 complexity

2014-01-10 Thread M.-A. Lemburg

On 09.01.2014 22:45, Antoine Pitrou wrote:
 On Thu, 9 Jan 2014 13:36:05 -0800
 Chris Barker chris.bar...@noaa.gov wrote:

 Some folks have suggested using latin-1 (or other 8-bit encoding) -- is
 that guaranteed to work with any binary data, and round-trip accurately?
 
 Yes, it is.

Just a word of caution:

Using the 'latin-1' to mean unknown encoding can easily result
in Mojibake (unreadable text) entering your application with
dangerous effects on your other text data.

E.g. Marc-André read using 'latin-1' if the string itself
is encoded as UTF-8 will give you Marc-AndrÃ© in your
application. (Yes, I see that a lot in applications
and websites I use ;-))

Also note that indexing based on code points will likely
break that way as well, ie. if you pass an index to an
application based on what you see in your editor or
shell, those indexes can be wrong when used on the
encoded data. UTF-8 is an example of a popular variable
length encoding for Unicode, so you'll hit this problem
whenever dealing with non-ASCII UTF-8 data.

 and will surrogateescape work for arbitrary binary data?
 
 Yes, it will.

The surrogateescape trick only works if you are encoding
your work using the same encoding that you used for decoding
it. Otherwise, you'll get a mix of the input encoding and the
output encoding as output.

Note that the error handler trick has an advantage over the
latin-1 trick: if you try to encode a Unicode string
with escape surrogates without using the error handler,
it will fail, so you at least know that there are funny
code points in your output string that need some extra
care.

BTW: Perhaps it would be a good idea to backport the
surrogateescape error handler to Python 2.7 to simplify
writing code which works in both Python 2 and 3.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 10 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5

2014-01-08 Thread M.-A. Lemburg

On 06.01.2014 14:24, Victor Stinner wrote:
 Hi,
 
 bytes % args and bytes.format(args) are requested by Mercurial and
 Twisted projects. The issue #3982 was stuck because nobody proposed a
 complete definition of the new features. Here is a try as a PEP.
 
 The PEP is a draft with open questions. First, I'm not sure that both
 bytes%args and bytes.format(args) are needed. The implementation of
 .format() is more complex, so why not only adding bytes%args?

+1 on doing all of this.

I'd simply copy over the Python 2 PyString code and start working
from there.

Readding these features makes live a lot easier in situations where you
have to work on data which is encoded text using multiple (sometimes
even unknown) encodings in a single data chunk. Think MIME messages,
mbox files, diffs, etc.

In such situations you often know the encoding of the part you're
working on (in most cases ASCII), but not necessarily the encodings
of other parts of the chunks.

You could work around this by decoding from Latin-1, then using Unicode
methods and encoding back to Latin-1, but the risk of letting Mojibake
enter your application in uncontrolled ways are high.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP process entry point and ill fated initiatives

2013-11-29 Thread M.-A. Lemburg

On 29.11.2013 16:25, Kristján Valur Jónsson wrote:
 How about acknowledging that these waters are dark and murky and help
 making things better?

 Well, how about? If Anatoly has a concrete proposal, surely he can propose a
 patch to make things better.

 Which is what  he did.  And instead of helpful ideas on how to improve his 
 patch,
 the issue is closed.  The acolytes have spoken.

I'm not sure we're talking about the same issue item. Anatoly did
not present a patch which could have been improved:

http://bugs.python.org/issue19822

Anatoly refuses to sign the Python CLA, so it's not surprising that he
didn't provide a patch.

His only contribution was a step-by-step process he put in the ticket
description which doesn't represent the PEP process. Christian pointed
him to the correct process, but he unfortunately just ignored this.

I'm afraid there's nothing much we can do to make ends meet.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 29 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add transform() and untranform() methods

2013-11-16 Thread M.-A. Lemburg

On 16.11.2013 01:47, Victor Stinner wrote:
 Adding transform()/untransform() method to bytes and str is a non
 trivial change and not everybody likes them. Anyway, it's too late for
 Python 3.4.

Just to clarify: I still like the idea of adding those methods.

I just don't see what this addition has to do with the codecs.encode()/
.decode() functions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 16 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-11-19: Python Meeting Duesseldorf ...  3 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the story on Py_FrozenMain?

2013-11-16 Thread M.-A. Lemburg

On 16.11.2013 18:48, Eric Snow wrote:
 While looking at something unrelated, I happened to peek at
 Python/frozenmain.c and found Py_FrozenMain().  I kind of get the idea
 of it, but am curious what motivated the addition and who might be
 using it.  The function is not documented and doesn't have much
 explanation.  I'm guessing that not many are familiar with it (e.g.
 http://bugs.python.org/issue15893).
 
 FWIW the function was added quite a while ago (and hasn't been touched
 a whole lot since):
 
 changeset:   1270:14369a5e61679364deeae9a9a0deedbd593a72e0
 branch:  legacy-trunk
 user:Guido van Rossum gu...@python.org
 date:Thu Apr 01 20:59:32 1993 +
 summary: Support for frozen scripts; added -i option.

It's used as main()-function for frozen Python interpreters.

See eGenix PyRun as an example and the freeze tool in Tools/freeze/
for the implementation that uses this API:

http://www.egenix.com/products/python/PyRun/

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 16 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-11-19: Python Meeting Duesseldorf ...  3 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add transform() and untranform() methods

2013-11-15 Thread M.-A. Lemburg

On 15.11.2013 08:13, Nick Coghlan wrote:
 On 15 November 2013 11:10, Terry Reedy tjre...@udel.edu wrote:
 On 11/14/2013 5:32 PM, Victor Stinner wrote:

 I don't like the functions codecs.encode() and codecs.decode() because
 the type of the result depends on the encoding (second parameter). We
 try to avoid this in Python.


 Such dependence is common with arithmetic.

 1 + 2
 3
 1 + 2.0
 3.0
 1 + 2+0j
 (3+0j)

 sum((1,2,3), 0)
 6
 sum((1,2,3), 0.0)
 6.0
 sum((1,2,3), 0.0+0j)
 (6+0j)

 for f in (compile, eval, getattr, iter, max, min, next, open, pow, round,
 type, vars):
   type(f(*args)) # depends on the inputs
 That is a large fraction of the non-class builtin functions.
 
 *Type* dependence between inputs and outputs is common (and completely
 non-controversial). The codecs system is different, since the
 supported input and output types are *value* dependent, driven by the
 name of the codec.
 
 That's the part which makes the codec machinery interesting in
 general, since it combines a value driven lazy loading mechanism
 (based on the codec name) with the subsequent invocation of that
 mechanism: the default codec search algorithm goes hunting in the
 encodings package (or the alias dictionary), but you can register
 custom search algorithms and provide encodings any way you want. It
 does mean, however, that the most you can claim for the type signature
 of codecs.encode and codecs.decode is that they accept an object and
 return an object. Beyond that, it's completely driven by the value of
 the codec.

Indeed. You have to think of the codec registry as a mere
lookup mechanism - very much like an import. The implementation
of the imported module defines which types are supported and
how the encode/decode steps work.

 In Python 2.x, the type constraints imposed by the str and unicode
 convenience methods is basestring in, basestring out. As it happens,
 all of the standard library codecs abide by that restriction , so it
 was easy to interpret the codecs module itself as having the same
 basestring in, basestring out limitation, especially given the heavy
 focus on text encodings in the way it was documented. In practice, the
 codecs weren't that open ended - some of them only accepted 8 bit
 strings, some only accepted unicode, some accepted both (perhaps
 relying on implicit decoding to unicode),
 
 The migration to Python 3 made the contrast between the two far more
 stark however, hence the long and involved discussion on issue 7475,
 and the fact that the non-Unicode codecs are currently still missing
 their shorthand aliases.
 
 The proposal I posted to issue 7475 back in April (and, in the absence
 of any objections to the proposal, finally implemented over the past
 few weeks) was to take advantage of the fact that the codecs.encode
 and codecs.decode convenience functions exist (and have been covered
 by the regression test suite) as far back as Python 2.4. I did this
 merely by documenting the existing of the functions for Python 2.7,
 3.3 and 3.4, changing the exception messages thrown for codec output
 type errors on the convenience methods to reference them, and by
 updating the Python 3.4 What's New document to explain the changes.
 
 This approach provides a Python 2/3 compatible solution for usage of
 non-Unicode encodings: users simply need to call the existing module
 level functions in the codecs module, rather than using the methods on
 specific builtin types. This approach also means that the binary
 codecs can be used with any bytes-like object (including memoryview
 and array.array), rather than being limited to types that implement a
 new method (like transform), and can also be used in Python 2/3
 source compatible APIs (since the data driven nature of the problem
 makes 2to3 unusable as a solution, and that doesn't help single code
 base projects anyway).

Right, and that was the main point in making codecs flexible
in this respect. There are many other types which can serve
as input and output - in the stdlib and interpreter as well as
in extension modules that implement their own types.

From my point of view, this is now just a matter of better documenting
 the status quo, and nudging people in the right direction when it
 comes to using the appropriate API for non-Unicode codecs. Since we
 now realise these functions have existed since Python 2.4, it doesn't
 make sense to try to fundamentally change direction, but instead to
 work on making it better.
 
 A few things I noticed while implementing the recent updates:
 
 - as you noted in your other email, while MAL is on record as saying
 the codecs module is intended for arbitrary codecs, not just Unicode
 encodings, readers of the current docs can definitely be forgiven for
 not realising that. We really need to better separate the codecs
 module docs from the text model docs (two new sections in the language
 reference, one for the codecs machinery and one for the text model
 would likely be appropriate. The io

Re: [Python-Dev] Add transform() and untranform() methods

2013-11-15 Thread M.-A. Lemburg

On 15.11.2013 12:45, Nick Coghlan wrote:
 On 15 November 2013 20:33, Antoine Pitrou solip...@pitrou.net wrote:
 On Fri, 15 Nov 2013 21:28:35 +1100
 Steven D'Aprano st...@pearwood.info wrote:

 One benefit is:

 import codecs
 codec = get_name_of_compression_codec()
 result = codecs.encode(data, codec)

 That's a good point.

 If encoding/decoding is intended to be completely generic (even if 99%
 of the uses will be with strings and bytes), is there any reason to
 prefer built-in functions rather than methods on object?

 Practicality beats purity. Personally, I've never used codecs on
 anything else than str and bytes objects.
 
 The reason I'm now putting some effort into better documenting the
 status quo for codec handling in Python 3 and filing off some of the
 rough edges (rather than proposing adding any new APIs to Python 3.x)
 is because the users I care about in this matter are web developers
 that already make use of the binary codecs and are adopting the
 single-source approach to handle supporting both Python 2 and Python
 3. Armin Ronacher is the one who's been most vocal about the problem,
 but he's definitely not alone.

You can add me to that list :-). Esp. the hex codec is very handy.
Google returns a few thousand hits for that codec alone.

One detail that people often tend to forget is the extensibility
of the codec system. It is easily possible to add new codecs
to the system to e.g. perform encoding, escaping, compression or
other conversion operations, so the set of codecs in the stdlib
is not the complete set of codecs used in the wild - and it's
not intended to be.

As example: We've written codecs for customers that perform
special types of XML un/escaping.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 15 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-11-19: Python Meeting Duesseldorf ...  4 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Restoring the aliases for the non-Unicode codecs

2013-11-13 Thread M.-A. Lemburg

On 13.11.2013 15:29, Nick Coghlan wrote:
 Back in Python 3.2, the non-Unicode codecs were restored to the
 standard library, but without the associated aliases (mostly due to
 some thoroughly confusing error messages when they were mistakenly
 used with the Unicode encoding convenience methods).
 
 The long gory history in http://bugs.python.org/issue7475 took a
 different turn earlier this year when I noticed
 (http://bugs.python.org/issue7475#msg187698) that the codecs module
 already *had* type neutral helper functions in the form of
 codecs.encode and codecs.decode, and has had them since Python 2.4.
 These were covered in the test suite, but not in the documentation.
 
 That realisation substantially changed my perspective on the issue,
 since it was no longer a matter of adding a new API, but of better
 documenting and facilitating the use of one we already had (and was
 supported all the way back to Python 2.4, make it easy to use in
 single-source Python 2/3 projects as well). Since then, three key
 supporting issues have been addressed for Python 3.4:
 
 * codecs.encode() and codecs.decode() have been documented in 2.7, 3.3
 and default (http://bugs.python.org/issue17827)
 * codec errors have been updated to incorporate the name of the codec
 whenever feasible, and output type errors in Unicode encoding
 convenience methods refer uses to the type neutral object-object
 convenience functions (http://bugs.python.org/issue17839)
 * the especially confusing errors from the base64 codecs have been
 eliminated by updating the base64 module to use memoryview to process
 binary input rather than explicit typechecks on builtin types
 (http://bugs.python.org/issue17839)
 
 So, with those underlying issues resolved, I would now like to restore
 the aliases for the non-Unicode codecs that were removed in
 http://bugs.python.org/issue10807 (aliases) and
 http://bugs.python.org/issue17841 (docs).

+1 and thanks for your work on this.

 I also looked into the possibility of providing appropriate 2to3
 fixers, but the data driven nature of the problem makes that
 impractical. However, the presence of codecs.decode and codecs.encode
 in Python 2.4+ makes a new Py3k warning in 2.7.7 a viable option:
 http://bugs.python.org/issue19543
 
 Regards,
 Nick.
 
 P.S. Until the next docs rebuild, you can see a summary of the
 non-Unicode codec handling improvements in the What's New diff here:
 http://hg.python.org/cpython/rev/854a2cea31b9
 
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 13 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-11-19: Python Meeting Duesseldorf ...  6 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Infrastructure] bugs.python.org not reachable in IPv6?

On 04.11.2013 16:10, Benjamin Peterson wrote:
 2013/11/4 M.-A. Lemburg m...@egenix.com:
 On 04.11.2013 11:01, Victor Stinner wrote:
 Hi,

 bugs.python.org is still not responding on IPv6. Can someone please
 remove the  record from python.org DNS, or fix the IPv6
 configuration?

 It's an issue on my PC because my PC has IPv6 address and so it cannot
 reach bugs.python.org. wget is really slow because it tries first
 IPv6, but then falls back to IPv4. Firefox tries first IPv4 and so
 doesn't have the issue.

 The infrastructure team cannot do anything about this, since
 the tracker is hosted by Upfront Systems.
 
 But the PSF controls the DNS, no?
 
 ;; AUTHORITY SECTION:
 python.org. 2396IN  NS  ns2.p11.dynect.net.
 python.org. 2396IN  NS  ns3.p11.dynect.net.
 python.org. 2396IN  NS  ns1.p11.dynect.net.
 python.org. 2396IN  NS  ns4.p11.dynect.net.

Yes, but only Upfront Systems knows about the correct IPv6 address :-)

Simply removing it would be short term work-around, though.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 04 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-11-19: Python Meeting Duesseldorf ... 15 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Infrastructure] bugs.python.org not reachable in IPv6?

On 04.11.2013 11:01, Victor Stinner wrote:
 Hi,
 
 bugs.python.org is still not responding on IPv6. Can someone please
 remove the  record from python.org DNS, or fix the IPv6
 configuration?
 
 It's an issue on my PC because my PC has IPv6 address and so it cannot
 reach bugs.python.org. wget is really slow because it tries first
 IPv6, but then falls back to IPv4. Firefox tries first IPv4 and so
 doesn't have the issue.

The infrastructure team cannot do anything about this, since
the tracker is hosted by Upfront Systems.

They are hosting the instance at Hetzner in Germany, which does
support IPv6.

Do we have someone with login permission for the server running
RoundUp ?

If not, we should probably find someone at Upfront Systems to check
the configuration.

 Victor
 
 2013/10/12 Antoine Pitrou anto...@python.org:

 Opened issue at http://psf.upfronthosting.co.za/roundup/meta/issue528

 Regards

 Antoine.



 Le samedi 12 octobre 2013 à 14:40 +0200, Victor Stinner a écrit :
 Hi,

 The DNS server of python.org announce the IP address 2a01:4f8:131:2480::3:

 $ host -t  bugs.python.org
 bugs.python.org has IPv6 address 2a01:4f8:131:2480::3


 The problem is that I cannot connect to this IP address:

 $ ping6 -c 4 2a01:4f8:131:2480::3
 PING 2a01:4f8:131:2480::3(2a01:4f8:131:2480::3) 56 data bytes

 --- 2a01:4f8:131:2480::3 ping statistics ---
 4 packets transmitted, 0 received, 100% packet loss, time 2999ms


 Do you have a the same issue, or is it just me?

 Victor
 
 Infrastructure mailing list
 infrastruct...@python.org
 https://mail.python.org/mailman/listinfo/infrastructure
 Unsubscribe: 
 https://mail.python.org/mailman/options/infrastructure/solipsis%40pitrou.net



 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
 
 Infrastructure mailing list
 infrastruct...@python.org
 https://mail.python.org/mailman/listinfo/infrastructure
 Unsubscribe: 
 https://mail.python.org/mailman/options/infrastructure/mal%40egenix.com
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 04 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-11-19: Python Meeting Duesseldorf ... 15 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Infrastructure] bugs.python.org not reachable in IPv6?

On 04.11.2013 17:05, R. David Murray wrote:
 On Mon, 04 Nov 2013 16:34:46 +0100, Antoine Pitrou anto...@python.org wrote:
 On lun., 2013-11-04 at 16:09 +0100, M.-A. Lemburg wrote:
 On 04.11.2013 11:01, Victor Stinner wrote:
 Hi,

 bugs.python.org is still not responding on IPv6. Can someone please
 remove the  record from python.org DNS, or fix the IPv6
 configuration?

 It's an issue on my PC because my PC has IPv6 address and so it cannot
 reach bugs.python.org. wget is really slow because it tries first
 IPv6, but then falls back to IPv4. Firefox tries first IPv4 and so
 doesn't have the issue.

 The infrastructure team cannot do anything about this, since
 the tracker is hosted by Upfront Systems.

 They are hosting the instance at Hetzner in Germany, which does
 support IPv6.

 Do we have someone with login permission for the server running
 RoundUp ?

 I think Ezio and Martin have access to the machine, but not necessarily
 root access; and, furthermore, the problem might be somewhere else on
 the network.

 I think the best short-term measure is to disable the IPv6 resolution
 for bugs.python.org. Otherwise people will run in the same issue over
 and over again.
 
 We have root, but I don't think that helps.
 
 I currently see this:
 
 eth0  Link encap:Ethernet  HWaddr 6c:62:6d:85:bb:7b  
   inet addr:46.4.197.70  Bcast:46.4.197.71  Mask:255.255.255.248
   inet6 addr: 2a01:4f8:131:2480::3/64 Scope:Global
 
 so the problem appears to be a network issue (as far as I can see,
 there's no firewall running on the box itself).  Disabling the AAA name
 record until we get a response from upfront sounds like a good idea.
 (I've added the upfront folks as nosy on the ticket Antoine opened
 on the meta-tracker.)

Some things to try on the box:

* ping6 2001:888:2000:d::a2 (that's python.org)

* check whether the web server is actually listening on the
  IPv6 address

* check whether a IPv6 default route is set up

If the first doesn't work, there's likely a router problem with
the server, which only Upfront can resolve (via a support request
to Hetzner or by checking their server IPv6 settings).

Here's the configuration page for the IPv6 setup:

http://wiki.hetzner.de/index.php/Netzkonfiguration_Debian/en

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 04 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-11-19: Python Meeting Duesseldorf ... 15 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Infrastructure] bugs.python.org not reachable in IPv6?