[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset c981ad16b0f9740bd3381c96b4227a1faa1a88d9 by Serhiy Storchaka (Jon 
Janzen) in branch 'master':
bpo-26707: Enable plistlib to read UID keys. (GH-12153)
https://github.com/python/cpython/commit/c981ad16b0f9740bd3381c96b4227a1faa1a88d9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Jon Janzen  added the comment:

I recently upgraded my python version and my hot-patch broke due to changes in 
bpo-32072 (GH-4455).

It reminded me of this b.p.o., and after reading through the messages to remind 
myself where the patch stood I realized that my tone friendly towards the end 
was not overly.

@ronaldoussoren, I apologize if I offended.

In other news, I re-implemented the patch (and filed a new pull-request) due to 
the following:

* Things got spread out over too many commits
* NSKeyedArchiver only uses binary mode, so I removed the XML compatibility
* I also wrote a few tests to verify the UID implementation

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Change by Jon Janzen :


--
pull_requests: +12152

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Jon Janzen  added the comment:

Ping

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Jon Janzen  added the comment:

Support for KeyedArchives are not limited to the Swift implementation I linked 
to. They have been supported since Mac OS X since 10.2 (long before Swift came 
around). The documentation 
(https://developer.apple.com/documentation/foundation/nskeyedarchiver?language=objc)
 shows that NSKeyedArchive can only output in plist format since outputFormat 
is of type NSPropertyListFormat (allowing to output in either XML or binary).

The other unimplemented binary token types (URL, UUID, set, ordset) are not 
used under NSKeyedArchive (see the "Encoding Data and Objects" section of the 
documentation mentioned above) so there's no concern that supporting 0x80 (UID) 
will suddenly necessitate implementing the other unimplemented types. If you 
feel that it would be necessary to implement them in order to accept the patch 
I would be happy to try and implement them.

I know I certainly have an use case (reading to-do list data from the 2Do app) 
and the creator of this bug wanted to read SnapChat data files.

Currently, I am using a hot-patched plistlib._BinaryPlistParser to read the 
data I need (see attached for a snippet) and I would rather not do that, but if 
you think my use case scope does not warrant inclusion in the standard library 
then I'll just have to deal with that.

--
Added file: https://bugs.python.org/file47483/plist_hack.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
type: behavior -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I'm still not too happy about supporting UIDs in plistlib, especially because 
I'm not sure it that's all that's needed. AFAIK I removed more types that the 
underlying encoder supported from plistlib because those are never used in 
plist files. 

The swift encoder for keyed archives is probably not the code that's actually 
used on the OS, AFAIK that still is Objective-C code.

P.S. I changed the version selection to 3.8, adding support for UIDs would be a 
feature change and not suited for back ports.

--
versions: +Python 3.8 -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Jon Janzen  added the comment:

Ping

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Jon Janzen  added the comment:

@serhiy.storchaka: I've implemented a UID wrapper class

I've also updated the parser and writer classes to support the UID wrapper. The 
implementations for reading/writing XML UID tags match the implementations 
given by Apple's plutil distributed with macOS:

UID(x) becomes {'CF$UID': int(x)}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

But they use the plist format for serialization (as plists theirself use the 
XML format).

https://github.com/apple/swift-corelibs-foundation/blob/master/Foundation/NSKeyedArchiver.swift

Direct support of keyed archives would be better to implement in third-party 
package. But we can provide the support for low-level operations.

For distinguishing UIDs from integers and for being able to create plist files 
containing UIDs we need a special purposed class plist.UID. It will be a 
simpler wrapper around int with few methods: __index__(), __repr__(), 
__reduce__().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Ronald Oussoren]

Ronald Oussoren  added the comment:

@bigfootjon: Cocoa keyed archives are not plist files.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Jon Janzen  added the comment:

Hello,

I have attached a file extracted from the database of the 2Do App for iOS and 
macOS. The file contains information about tags used in the app.

plistlib cannot currently parse this file because it lacks the ability to read 
byte 0x80 (UID).

I believe the documentation for generating these type of files can be found at: 
https://developer.apple.com/documentation/foundation/nskeyedarchiver

--
nosy: +bigfootjon
Added file: https://bugs.python.org/file47468/cat.plist

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Note that I'm still -1 on merging this patch because it is unclear how to 
create such plist files using public Apple APIs.

P.S. The low-level code for creating and reading binary plist files appears to 
be used for more than juist plist archives.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +5693

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The size of UID data is just tokenL + 1, not self._get_size(tokenL + 1).

FYI, the code for support UIDs and other types was proposed in issue14455, but 
was excluded from the final patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Ronald Oussoren]

Ronald Oussoren added the comment:

How can you create plist files that contain UID values using Apple's APIs?

The plist library is meant to be interoperable with files created using Apple's 
APIs for creating plist files, and I didn't find an API that created UID values 
at the time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[John Lehr]

John Lehr added the comment:

Based on the format specification pointed to by Serhiy, perhaps this a better 
patch, correcting size from previous patch submission and treating:

706,707c706,708
< # tokenH == 0x80 is documented as 'UID' and appears to be used for
< # keyed-archiving, not in plists.
---
> elif tokenH == 0x80:  # UID
> s = self._get_size(tokenL + 1)
> return int.from_bytes(self._fp.read(s), 'big')

I have compared output with OS X plutil and plistlib.load() with this patch and 
the values are identical for UID fields.

--
Added file: http://bugs.python.org/file42395/plistlib_uid.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[John Lehr]

John Lehr added the comment:

I’m sorry, but the files in which I detected the problem cannot be circulated.  
I will try to create a test account on Snapchat and generate some test data, 
but I can’t do this anytime soon.

> On Apr 7, 2016, at 12:51 AM, SilentGhost  wrote:
> 
> 
> SilentGhost added the comment:
> 
> Here is the version of the patch suitable for the Rietveld. John, could you 
> perhaps provide an example file that uses UID values?
> 
> Also, the code is identical to handling of 0x50 token, perhaps it could be 
> incorporated into it.
> 
> --
> nosy: +SilentGhost
> stage:  -> patch review
> versions: +Python 3.6
> Added file: http://bugs.python.org/file42390/issue26707.diff
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[John Lehr]

John Lehr added the comment:

I’m glad you found it in the Apple specification.  I looked, but missed it.  I 
would absolutely defer to you on your assessment of the decoding.

> On Apr 7, 2016, at 1:46 AM, Serhiy Storchaka  wrote:
> 
> 
> Serhiy Storchaka added the comment:
> 
> UID is rather int than bytes. And I would use a special UID type.
> 
> According to Apple's sources [1], the size of UID data is tokenL+1, not 
> self._get_size(tokenL).
> 
> [1] http://www.opensource.apple.com/source/CF/CF-1153.18/CFBinaryPList.c
> 
> --
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

UID is rather int than bytes. And I would use a special UID type.

According to Apple's sources [1], the size of UID data is tokenL+1, not 
self._get_size(tokenL).

[1] http://www.opensource.apple.com/source/CF/CF-1153.18/CFBinaryPList.c

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[SilentGhost]

SilentGhost added the comment:

Here is the version of the patch suitable for the Rietveld. John, could you 
perhaps provide an example file that uses UID values?

Also, the code is identical to handling of 0x50 token, perhaps it could be 
incorporated into it.

--
nosy: +SilentGhost
stage:  -> patch review
versions: +Python 3.6
Added file: http://bugs.python.org/file42390/issue26707.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +ronaldoussoren, serhiy.storchaka
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[John Lehr]

New submission from John Lehr:

libplist raises an invalid file exception on loading properly formed binary 
plists containing UID (0x80) values.  The binary files were tested for form 
with plutil.

Comments at line 706 state the value is defined but not in use in plists, and 
the object is not handled.  However, I find them frequently in bplists, e.g., 
iOS Snapchat application files.  I have attached a proposed patch that I have 
tested on these files and can now successfully parse them with the _read_object 
method in the _BinaryPlistParser class.

My proposed patch is pasted below for others consideration while waiting for 
the issue to be resolved.

706,707c706,708
< # tokenH == 0x80 is documented as 'UID' and appears to be used for
< # keyed-archiving, not in plists.
---
> elif tokenH == 0x80: #UID
> s = self._get_size(tokenL)
> return self._fp.read(s).decode('ascii')

Thanks for your consideration.

--
components: Library (Lib)
files: plistlib_uid.diff
keywords: patch
messages: 262974
nosy: slo.sleuth
priority: normal
severity: normal
status: open
title: plistlib fails to parse bplist with 0x80 UID values
type: crash
versions: Python 3.5
Added file: http://bugs.python.org/file42388/plistlib_uid.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com