On 17.03.2016 01:29, Guido van Rossum wrote:
> I've updated the PEP. Please review. I decided not to update the
> Unicode howto (the thing is too obscure). Serhiy, you're probably in a
> better position to fix the code looking for cookies to pick the first
> one if there are two on the same line (or do whatever you think should
> be done there).

Thanks, will do.

> Should we recommend that everyone use tokenize.detect_encoding()?

I'd prefer a separate utility for this somewhere, since
tokenize.detect_encoding() is not available in Python 2.

I've attached an example implementation with tests, which works
in Python 2.7 and 3.

> On Wed, Mar 16, 2016 at 5:05 PM, Guido van Rossum <gu...@python.org> wrote:
>> On Wed, Mar 16, 2016 at 12:59 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>> The only reason to read up to two lines was to address the use of
>>> the shebang on Unix, not to be able to define two competing
>>> source code encodings :-)
>> I know. I was just surprised that the PEP was sufficiently vague about
>> it that when I found that mypy picked the second if there were two, I
>> couldn't prove to myself that it was violating the PEP. I'd rather
>> clarify the PEP than rely on the reasoning presented earlier here.

I suppose it's a rather rare case, since it's the first time
that I heard about anyone thinking that a possible second line
could be picked - after 15 years :-)

>> I don't like erroring out when there are two different cookies on two
>> lines; I feel that the spirit of the PEP is to read up to two lines
>> until a cookie is found, whichever comes first.
>> I will update the regex in the PEP too (or change the wording to avoid 
>> "match").
>> I'm not sure what to do if there are two cooking on one line. If
>> CPython currently picks the latter we may want to preserve that
>> behavior.
>> Should we recommend that everyone use tokenize.detect_encoding()?
>> --
>> --Guido van Rossum (python.org/~guido)

Marc-Andre Lemburg

Professional Python Services directly from the Experts (#1, Mar 17 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89
2016-02-19: Released eGenix PyRun 2.1.2 ...       http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

    Utility to detect the source code encoding of a Python file.

    Marc-Andre Lemburg, 2016.

    Supports Python 2.7 and 3.

import sys
import re
import codecs

# Debug output ?
_debug = True

# PEP 263 RE
PEP263 = re.compile(b'^[ \t]*#.*?coding[:=][ \t]*([-.a-zA-Z0-9]+)',


def detect_source_encoding(code, buffer_size=400):

    """ Detect and return the source code encoding of the Python code
        given in code.
        code must be given as bytes.
        The function uses a buffer to determine the first two code lines
        with a default size of 400 bytes/code points.  This can be adjusted
        using the buffer_size parameter.
    # Get the first two lines
    first_two_lines = b'\n'.join(code[:buffer_size].splitlines()[:2])
    # BOMs override any source code encoding comments
    if first_two_lines.startswith(codecs.BOM):
        return 'utf-8'
    # .search() picks the first occurrance
    m = PEP263.search(first_two_lines)
    if m is None:
        return 'ascii'
    return m.group(1).decode('ascii')

# Tests

def _test():

    l = (
# No encoding
""", 'ascii'),
# coding: latin-1
""", 'latin-1'),
# coding: utf-8
""", 'utf-8'),
# The above could be detected as source code encoding
""", 'ascii'),
# coding: latin-1
# coding: utf-8
""", 'latin-1'),
# No encoding on first line
# No encoding on second line
# coding: utf-8
""", 'ascii'),
  (codecs.BOM + b"""\
# No encoding
""", 'utf-8'),
  (codecs.BOM + b"""\
# BOM and encoding
# coding: latin-1
""", 'utf-8'),
    for code, encoding in l:
        if _debug:
            print ('=' * 72)
            print ('Checking:')
            print ('-' * 72)
            print (code.decode('latin-1'))
            print ('-' * 72)
        detected_encoding = detect_source_encoding(code)
        if _debug:
            print ('detected: %s, expected: %s' % 
                   (detected_encoding, encoding))
        assert detected_encoding == encoding

if __name__ == '__main__':
Python-Dev mailing list

Reply via email to