Re: [Python-Dev] bitfields - short - and xlc compiler

2016-03-20 Thread Terry Reedy 

On 3/20/2016 4:04 PM, Andrew Barnert via Python-Dev wrote:


Agreed. But I think the test is reasonable on at least MSVC, gcc, clang, and 
icc. So what you need is some way to run the test on those compilers, but not 
on compilers that can't handle it.


The test could be conditioned on the compiler used.

>>> platform.python_compiler()
'MSC v.1900 64 bit (AMD64)'

The doc could say something about the feature only being available with 
certain compilers.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitfields - short - and xlc compiler

2016-03-20 Thread Andrew Barnert via Python-Dev 
On Mar 20, 2016, at 09:07, Michael Felt  wrote:
> 
>> On 2016-03-18 05:57, Andrew Barnert via Python-Dev wrote:
>> Yeah, C99 (6.7.2.1) allows "a qualified or unqualified version of _Bool, 
>> signed int, unsigned int, or some other implementation-defined type", and 
>> same for C11. This means that a compiler could easily allow an 
>> implementation-defined type that's identical to and interconvertible with 
>> short, say "i16", to be used in bitfields, but not short itself.
>> 
>> And yet, gcc still allows short "even in strictly conforming mode" (4.9), 
>> and it looks like Clang and Intel do the same.
>> 
>> Meanwhile, MSVC specifically says it's illegal ("The type-specifier for the 
>> declarator must be unsigned int, signed int, or int") but then defines the 
>> semantics (you can't have a 17-bit short, bit fields act as the underlying 
>> type when accessed, alignment is forced to a boundary appropriate for the 
>> underlying type). They do mention that allowing char and long types is a 
>> Microsoft extension, but still nothing about short, even though it's used in 
>> most of the examples on the page.
>> 
>> Anyway, is the question what ctypes should do? If a platform's compiler 
>> allows "short M: 1", especially if it has potentially different alignment 
>> than "int M: 1", ctypes on that platform had better make ("M", c_short, 1) 
>> match the former, right?
>> 
>> So it sounds like you need some configure switch to test that your compiler 
>> doesn't allow short bit fields, so your ctypes build at least skips that 
>> part of _ctypes_test.c and test_bitfields.py, and maybe even doesn't allow 
>> them in Python code.
>> 
>> 
 >>  test_short fails om AIX when using xlC in any case. How terrible is 
 >> this?

> a) this does not look solveable using xlC, and I expect from the comment 
> above re: MSVC, that it will, or should also fail there.

> And, imho, if anything is to done, it is a decision to be made by "Python".

Sure, but isn't that exactly why you're posting to this list?

> b) aka - it sounds like a defect, at least in the test.

Agreed. But I think the test is reasonable on at least MSVC, gcc, clang, and 
icc. So what you need is some way to run the test on those compilers, but not 
on compilers that can't handle it.

So it sounds like you need a flag coming from autoconf that can be tested in C 
(and probably in Python as well) that tells you whether the compiler can handle 
it. And I don't think there is any such flag. 

Which means someone would have to add the configure test. And if people who use 
MSVC, gcc, and clang are all unaffected, I'm guessing that someone would have 
to be someone who cares about xlC or some other compiler, like you.

The alternative would be to just change the docs to make it explicit that using 
non-int bitfields isn't supported but may work in platform-specific ways. If 
you got everyone to agree to that, surely you could just remove the tests, 
right? But if people are actually writing C code that follows the examples on 
the MSVC bitfield docs page, and need to talk to that code from ctypes, I don't 
know if it would be acceptable to stop officially supporting that.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting references to Py_None

2016-03-20 Thread Tim Peters 
[Facundo Batista ]
> I'm seeing that our code increases the reference counting to Py_None,
> and I find this a little strange: isn't Py_None eternal and will never
> die?

Yes, but it's immortal in CPython because its reference count never
falls to 0 (it's created with a reference count of 1 to begin with).
That's the only thing that makes it immortal.


> What's the point of counting its references?

Uniformity and simplicity:  code using a PyObject* increments and
decrements reference counts appropriately with no concern for what
_kind_ of object is being pointed at.  All objects (including None)
are treated exactly the same way for refcount purposes.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting references to Py_None

2016-03-20 Thread Brett Cannon 
On Sun, 20 Mar 2016 at 09:44 Facundo Batista 
wrote:

> Hello!
>
> I'm seeing that our code increases the reference counting to Py_None,
> and I find this a little strange: isn't Py_None eternal and will never
> die?
>

Semantically yes, but we have to technically make that happen. :)


>
> What's the point of counting its references?
>

It's just another Python object, so if you return it then the code that
receives it may very well decref it because it always DECREFs the object
returned. And if we didn't keep its count accurately it would eventually
hit zero and constantly have its dealloc function checked for. We could add
a magical "never dies" value for the refcount, but that adds another `if`
branch in all the code instead of simply treating Py_None like any other
object and properly tracking its  reference.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting references to Py_None

2016-03-20 Thread David Wilson 
On Sun, Mar 20, 2016 at 01:43:27PM -0300, Facundo Batista wrote:
> Hello!
> 
> I'm seeing that our code increases the reference counting to Py_None,
> and I find this a little strange: isn't Py_None eternal and will never
> die?
> 
> What's the point of counting its references?

Avoiding a branch on every single Py_DECREF / Py_INCREF?

> 
> Thanks!
> 
> -- 
> .Facundo
> 
> Blog: http://www.taniquetil.com.ar/plog/
> PyAr: http://www.python.org/ar/
> Twitter: @facundobatista
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/dw%2Bpython-dev%40hmmz.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Counting references to Py_None

2016-03-20 Thread Facundo Batista 
Hello!

I'm seeing that our code increases the reference counting to Py_None,
and I find this a little strange: isn't Py_None eternal and will never
die?

What's the point of counting its references?

Thanks!

-- 
.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
Twitter: @facundobatista
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitfields - short - and xlc compiler

2016-03-20 Thread Michael Felt 



On 2016-03-18 05:57, Andrew Barnert via Python-Dev wrote:

Yeah, C99 (6.7.2.1) allows "a qualified or unqualified version of _Bool, signed int, unsigned 
int, or some other implementation-defined type", and same for C11. This means that a compiler 
could easily allow an implementation-defined type that's identical to and interconvertible with 
short, say "i16", to be used in bitfields, but not short itself.

And yet, gcc still allows short "even in strictly conforming mode" (4.9), and 
it looks like Clang and Intel do the same.

Meanwhile, MSVC specifically says it's illegal ("The type-specifier for the 
declarator must be unsigned int, signed int, or int") but then defines the semantics 
(you can't have a 17-bit short, bit fields act as the underlying type when accessed, 
alignment is forced to a boundary appropriate for the underlying type). They do mention 
that allowing char and long types is a Microsoft extension, but still nothing about 
short, even though it's used in most of the examples on the page.

Anyway, is the question what ctypes should do? If a platform's compiler allows "short M: 1", 
especially if it has potentially different alignment than "int M: 1", ctypes on that platform had 
better make ("M", c_short, 1) match the former, right?

So it sounds like you need some configure switch to test that your compiler 
doesn't allow short bit fields, so your ctypes build at least skips that part 
of _ctypes_test.c and test_bitfields.py, and maybe even doesn't allow them in 
Python code.



>>  test_short fails om AIX when using xlC in any case. How terrible is this?
a) this does not look solveable using xlC, and I expect from the comment 
above re: MSVC, that it will, or should also fail there. And, imho, if 
anything is to done, it is a decision to be made by "Python".

b) aka - it sounds like a defect, at least in the test.
c) what danger is there to existing Python code if "short" is expected, 
per legacy when compilers did (and GCC still does - verified that when I 
compile with gcc the test does not signal failure)


So, more with regard to c) - is there something I could/should be 
looking at in Python itself, in order to message that the code is not 
supported by the compiler?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-20 Thread Guido van Rossum 
On Thu, Mar 17, 2016 at 9:50 AM, Serhiy Storchaka  wrote:
> On 17.03.16 16:55, Guido van Rossum wrote:
>>
>> On Thu, Mar 17, 2016 at 5:04 AM, Serhiy Storchaka 
>> wrote:

 Should we recommend that everyone use tokenize.detect_encoding()?
>>>
>>>
>>> Likely. However the interface of tokenize.detect_encoding() is not very
>>> simple.
>>
>>
>> I just found that out yesterday. You have to give it a readline()
>> function, which is cumbersome if all you have is a (byte) string and
>> you don't want to split it on lines just yet. And the readline()
>> function raises SyntaxError when the encoding isn't right. I wish
>> there were a lower-level helper that just took a line and told you
>> what the encoding in it was, if any. Then the rest of the logic can be
>> handled by the caller (including the logic of trying up to two lines).
>
>
> The simplest way to detect encoding of bytes string:
>
> lines = data.splitlines()
> encoding = tokenize.detect_encoding(iter(lines).__next__)[0]

This will raise SyntaxError if the encoding is unknown. That needs to
be caught in mypy's case and then it needs to get the line number from
the exception. I tried this and it was too painful, so now I've just
changed the regex that mypy uses to use non-eager matching
(https://github.com/python/mypy/commit/b291998a46d580df412ed28af1ba1658446b9fe5).

> If you don't want to split all data on lines, the most efficient way in
> Python 3.5 is:
>
> encoding = tokenize.detect_encoding(io.BytesIO(data).readline)[0]
>
> In Python 3.5 io.BytesIO(data) has constant complexity.

Ditto with the SyntaxError though.

> In older versions for detecting encoding without copying data or splitting
> all data on lines you should write line iterator. For example:
>
> def iterlines(data):
> start = 0
> while True:
> end = data.find(b'\n', start) + 1
> if not end:
> break
> yield data[start:end]
> start = end
> yield data[start:]
>
> encoding = tokenize.detect_encoding(iterlines(data).__next__)[0]
>
> or
>
> it = (m.group() for m in re.finditer(b'.*\n?', data))
> encoding = tokenize.detect_encoding(it.__next__)
>
> I don't know what approach is more efficient.

Having my own regex was simpler. :-(

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-20 Thread Ethan Furman 

On 03/17/2016 04:54 PM, Glenn Linderman wrote:

On 3/16/2016 12:59 AM, Serhiy Storchaka wrote:



Actually "must match the regular expression" is not correct, because
re.match() implies anchoring at the start. I have proposed more
correct regular expression in other branch of this thread.


"match" doesn't imply anchoring at the start.  "re.match()" does (and as
a result is very confusing to newbies to Python re, that have used other
regexp systems).


It still confuses me from time to time.  :(

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-20 Thread Nick Coghlan 
On 20 March 2016 at 07:46, Glenn Linderman  wrote:
> Diagnosing ambiguous conditions, even including my example above, might be
> useful... for a few files... is it worth the effort? What % of .py sources
> have coding specifications? What % of those have two?

And there's a decent argument for leaving detecting such cases to
linters rather than the tokeniser.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-20 Thread M.-A. Lemburg 
On 17.03.2016 15:02, Serhiy Storchaka wrote:
> On 17.03.16 15:14, M.-A. Lemburg wrote:
>> On 17.03.2016 01:29, Guido van Rossum wrote:
>>> Should we recommend that everyone use tokenize.detect_encoding()?
>>
>> I'd prefer a separate utility for this somewhere, since
>> tokenize.detect_encoding() is not available in Python 2.
>>
>> I've attached an example implementation with tests, which works
>> in Python 2.7 and 3.
> 
> Sorry, but this code doesn't match the behaviour of Python interpreter,
> nor other tools. I suggest to backport tokenize.detect_encoding() (but
> be aware that the default encoding in Python 2 is ASCII, not UTF-8).

Yes, I got the default for Python 3 wrong. I'll fix that. Thanks
for the note.

What other aspects are different than what Python implements ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Mar 17 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89
2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com