New submission from Aaron Staley <usaa...@gmail.com>:
The json library's encoder includes a function called 'iterencode'. iterencode
allows for encoding to be streamed; as tokens are produced they are yielded.
This allows for the encoded object to be streamed to a file, over a socket,
etc. without being placed all into memory.
Unfortunately, iterencode cannot encode general iterators. This significantly
limits the usefulness of the function. For my use case I wish to convert a
large stream (iterator) of objects into json. Unfortunately, I currently have
to:
A. Bring all the objects into memory by encasing the iterator in a list()
B. Make a hack where I subclass list and making that object's __iter__ function
return my desired iterator.
The problem is that the json library explicitly checks for something being a
list:
if isinstance(value, (list, tuple)):
chunks = _iterencode_list(value, _current_indent_level)
It would work just as well (and be more pythonic) to see if the value supports
the iterator protocol:
if isinstance(value, collections.Iterable):
chunks = _iterencode_list(value, _current_indent_level)
Erroring example:
>>> import json
>>> e = json.JSONEncoder()
>>> r = xrange(20)
>>> gen = e.iterencode(r)
<generator object _iterencode at 0x14a5460>
>>> next(gen)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.2/json/encoder.py", line 419, in _iterencode
o = _default(o)
File "/usr/lib/python3.2/json/encoder.py", line 170, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: xrange(0, 20) is not JSON serializable
----------
components: Library (Lib)
messages: 158239
nosy: Aaron.Staley
priority: normal
severity: normal
status: open
title: json iterencode can not handle general iterators
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue14573>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
