[issue21071] struct.Struct.format is bytes, but should be str

[STINNER Victor]

STINNER Victor added the comment:

Ok, I changed struct.Struct.format type to str (Unicode string).

If someone wants to modify the C code to use a PyUnicodeObject rather than a 
char*, feel free to propose a further change.

Since the initial issue is fixed, I now close the issue.

Thank you all for your feedback and reviews ;-)

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed
versions: +Python 3.7 -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[STINNER Victor]

STINNER Victor added the comment:


New changeset f87b85f80853c580b1c8bf78a51b0e9a25f6e1a7 by Victor Stinner in 
branch 'master':
bpo-21071: struct.Struct.format type is now str (#845)
https://github.com/python/cpython/commit/f87b85f80853c580b1c8bf78a51b0e9a25f6e1a7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Martin Panter]

Martin Panter added the comment:

I don’t think the API should be expanded to accept arbitrary bytes-like objects 
as format strings. Struct formats are strings of ASCII-compatible characters, 
but not arbitrary chunks of memory.

I think the main question is whether it is okay to break compatibility 
(Victor’s pull request, or my format-str.patch), or whether there has to be a 
backwards-compatible deprecation of the existing bytes attribute. FWIW I am 
okay with breaking compatibility, since the main documentation already implies 
it should be a text string.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Xiang Zhang]

Xiang Zhang added the comment:

The warnings are possible to remove I think... but deprecate bytes arguments 
sounds good.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

After changing the type of Struct.format to str we perhaps should deprecate 
accepting bytes as format. Currently this can lead to emitting a BytesWarning.

$ ./python -Wa -b
>>> import struct
>>> struct.pack('I', 12345)
b'90\x00\x00'
>>> struct.pack(b'I', 12345)
__main__:1: BytesWarning: Comparison between bytes and string
__main__:1: BytesWarning: Comparison between bytes and string
b'90\x00\x00'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Xiang Zhang]

Xiang Zhang added the comment:

+1 for change bytes to str. But struct.Struct() accepts both bytes and str, 
maybe in future buffer objects. When it gets a bytes object, converting it to a 
str looks unnecessary to me, and as OP said, comparison (a theoretical use 
case) could still fail. Could we just leave what the user passes in? 
bytes(bytes-like) -> bytes, str -> str. This looks more natural to me.

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[STINNER Victor]

STINNER Victor added the comment:

Ok, I opened a thread on python-dev:
https://mail.python.org/pipermail/python-dev/2017-March/147688.html

Martin: "At a minimum you should acknowledge it in the “porting”
section of What’s New."

I wasn't sure if the change was worth it to be mentionned in What's
New in Python 3.7. Ok, will do for the next round (I'm now waiting for
more feedback on my python-dev thread and this issue.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Martin Panter]

Martin Panter added the comment:

Hi Victor, I’m not sure about changing the data type. As Python 3 grows older, 
there is potentially more code being written that you break by fixing a bug 
like this. It is incompatible if you used to write

>>> print(struct.Struct('hi').format.decode())
hi

I have used this decode() trick in the past to build composite format strings; 
e.g.: . If you change the data 
type this code will raise AttributeError. At a minimum you should acknowledge 
it in the “porting” section of What’s New.

Also, if you make this change, maybe update the module doc string. See the end 
of format-str.patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This should be discussed on Python-Dev first. I already raised this issue on 
Python-Dev, but don't remember what is the result.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[STINNER Victor]

STINNER Victor added the comment:

I created https://github.com/python/cpython/pull/845 to change 
struct.Struct.format type to str (Unicode).

struct.Struct() accepts bytes and str format strings, so it's not really a 
backward incompatible change.

It's just a minor enhancement to help development:

$ ./python
Python 3.7.0a0 (heads/master-dirty:b8a7daf, Mar 27 2017, 13:02:20) 
>>> print(struct.Struct('hi').format)
hi


Without the patch:

haypo@selma$ python3
Python 3.5.2 (default, Sep 14 2016, 11:28:32) 
[GCC 6.2.1 20160901 (Red Hat 6.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import struct
>>> print(struct.Struct('hi').format)
b'hi'

haypo@selma$ python3 -bb
Python 3.5.2 (default, Sep 14 2016, 11:28:32) 
>>> import struct
>>> print(struct.Struct('hi').format)
Traceback (most recent call last):
  ...
BytesWarning: str() on a bytes instance

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +743

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Martin Panter]

Martin Panter added the comment:

A backwards-compatible way forward would be to preserve (and document) the 
“format” attribute as a byte string, and add a new attribute which is 
definitely a text string. Not sure of a good name; perhaps “Struct.text_format” 
or “format_str” is a start.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Raymond Hettinger]

Raymond Hettinger added the comment:

I would like to see this and issue 8934 discussed as a usability bug.  As far 
as I can tell, the current state of affairs an unintended by-product of a 
rushed effort to split the standard library to bytes apis and unicode apis.  I 
don't see any reason that we should have to live with this forever.

--
nosy: +rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[STINNER Victor]

STINNER Victor added the comment:

 Is it safe to assume PyUnicode_AsUTF8() is null-terminated?

Yes, Python ensures that the string is null terminated.

 (like PyBytes_AS_STRING() is)

Yes, PyBytes_AS_STRING() also ends with a null byte.

By the way, Unicode strings internally ends with a null character.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I think breaking the compatibility should be discussed on Python-Dev. Similar 
issue (and even worse) is issue8934.

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[R. David Murray]

R. David Murray added the comment:

A backward compatibility break would certainly need to be discussed, IMO.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[STINNER Victor]

STINNER Victor added the comment:

 It would be backwards incompatible to change, though.

I'm in favor of breaking the compatibility with Python 3.4 and return the 
format as an Unicode string.

--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Martin Panter]

Martin Panter added the comment:

I originally assumed it would be a text string from the existing documentation, 
so changing the behaviour to match also seems reasonable

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Martin Panter]

Martin Panter added the comment:

Here is a patch that changes over to a str() type.

Is it safe to assume PyUnicode_AsUTF8() is null-terminated (like 
PyBytes_AS_STRING() is)? My documentation doesn’t say.

--
Added file: http://bugs.python.org/file37488/format-str.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Martin Panter]

Martin Panter added the comment:

It seems to me that the simplest fix is to document:

1. Struct.format attribute is a byte string
2. The input format strings for struct.pack(), Struct class, etc, are also 
allowed to be byte strings, for consistency (Issue 16349)

Here is a patch that does that, and adds some simple test cases.

--
keywords: +patch
Added file: http://bugs.python.org/file37474/format-bytes.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Martin Panter]

Martin Panter added the comment:

This is closely related to Issue 16349. If format strings were explicitly 
allowed to be byte strings there would be less conflict, but documenting the 
data type of the “format” attribute is better than nothing.

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Zbyszek Jędrzejewski-Szmek]

New submission from Zbyszek Jędrzejewski-Szmek:

In Python 2, Struct.format used to be a str. In Python 3 it is bytes, which is 
unexpected.

Why do I expect .format to be a string:
- This format is pretty much the same as a {}-format - plain text
- according to documentation it is composed of things like characters from a 
closed set '.=@hi...', a subset of ASCII,
- it is always called format string in the documentation

Why is this a problem:
- If I use a str format in constructor, I expect to get a str format,
- Comparisons are broken:

 struct.Struct('x').format == 'x'
False
 struct.Struct('x').format[0] == 'x'
False

- doctests are broken
 struct.Struct('x').format
'x' # in Python 2
b'x' # in Python 3

--
components: Library (Lib)
messages: 214903
nosy: zbysz
priority: normal
severity: normal
status: open
title: struct.Struct.format is bytes, but should be str
type: behavior
versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Benjamin Peterson]

Benjamin Peterson added the comment:

I agree that's rather unfortunate. It would be backwards incompatible to 
change, though.

--
nosy: +benjamin.peterson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[Zbyszek Jędrzejewski-Szmek]

Zbyszek Jędrzejewski-Szmek added the comment:

Maybe a flag param for the constructor?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21071] struct.Struct.format is bytes, but should be str

[R. David Murray]

R. David Murray added the comment:

I agree that the implementation does not match the documentation in this case.  
Especially the part about the format string used to create this Struct 
object.  I don't see what having a flag would buy you: it doesn't help you in 
writing 2/3 shared code.  I think the best we can do here is a doc change.

--
assignee:  - docs@python
components: +Documentation
nosy: +docs@python, r.david.murray
stage:  - needs patch
versions:  -Python 3.1, Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21071
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com