[issue34204] Bump the default pickle protocol in shelve

[Cheryl Sabella]

Change by Cheryl Sabella :


--
nosy:  -miss-islington
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[miss-islington]

miss-islington  added the comment:


New changeset df59273c7a384ea8c890fa8e9b80c92825df841c by Zackery Spytz in 
branch 'master':
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
https://github.com/python/cpython/commit/df59273c7a384ea8c890fa8e9b80c92825df841c


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[Zackery Spytz]

Zackery Spytz  added the comment:

It has not been abandoned.

--
versions: +Python 3.10 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[Marco Castelluccio]

Marco Castelluccio  added the comment:

I've opened https://github.com/python/cpython/pull/22751 to fix this, I know 
there was already a PR, but it seems to have been abandoned.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[Marco Castelluccio]

Change by Marco Castelluccio :


--
nosy: +marco-c
nosy_count: 6.0 -> 7.0
pull_requests: +21928
pull_request: https://github.com/python/cpython/pull/22751

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[STINNER Victor]

STINNER Victor  added the comment:

I wrote a short script to see the impact of file size depending on the protocol:
---
import shelve
import os.path

print("== Short value ==")
for proto in (0, 1, 2, 3, 4, 5):
filename = 'shelve-picklev%s' % proto
with shelve.open(filename, protocol=proto) as db:
assert db._protocol == proto
for x in range(1000):
db[str(x)] = str(x)
print(f'Protocol {proto}: {os.path.getsize(filename)} bytes')
os.unlink(filename)
print()
print("== Large value ==")
large_value = [str(x) for x in range(1000)]
for proto in (0, 1, 2, 3, 4, 5):
filename = 'shelve-picklev%s' % proto
with shelve.open(filename, protocol=proto) as db:
assert db._protocol == proto
for x in range(10):
db[str(x)] = large_value
print(f'Protocol {proto}: {os.path.getsize(filename)} bytes')
os.unlink(filename)
---

Output with Python 3.9.0b1 (on Fedora 32):
---
== Short value ==
Protocol 0: 90112 bytes
Protocol 1: 94208 bytes
Protocol 2: 94208 bytes
Protocol 3: 94208 bytes
Protocol 4: 94208 bytes
Protocol 5: 94208 bytes

== Large value ==
Protocol 0: 139264 bytes
Protocol 1: 139264 bytes
Protocol 2: 139264 bytes
Protocol 3: 139264 bytes
Protocol 4: 98304 bytes
Protocol 5: 98304 bytes
---

For short string values, protocol 0 produces smaller files than protocol 1 and 
higher.

For large value, protocol 4 and higher produce smaller files than protocol 3 
and lower.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[Zackery Spytz]

Change by Zackery Spytz :


--
keywords: +patch
nosy: +ZackerySpytz
nosy_count: 4.0 -> 5.0
pull_requests: +18963
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19639

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34204] Bump the default pickle protocol in shelve

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

The default pickle protocol is 4 now. But shelve still uses the pickle protocol 
3. Shouldn't it be bumped? Shouldn't shelve use pickle.DEFAULT_PROTOCOL by 
default?

Disadvantages:

1. This will make shelve files incompatible with Python 3.3 by default.

2. Protocol 4 adds 9 bytes of overhead in comparison with protocol 3. This can 
be too large for the shelve containing a lot of small objects. Maybe strip 
redundant frame header for small pickles?

--
components: Library (Lib)
messages: 322281
nosy: alexandre.vassalotti, lukasz.langa, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Bump the default pickle protocol in shelve
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com