[
https://issues.apache.org/jira/browse/ARROW-11371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271325#comment-17271325
]
Joris Van den Bossche commented on ARROW-11371:
-----------------------------------------------
Note that the pyarrow.serialize functionality is deprecated: see
https://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization
> [Python] Serializing deeply nested structures raises recursion error
> --------------------------------------------------------------------
>
> Key: ARROW-11371
> URL: https://issues.apache.org/jira/browse/ARROW-11371
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 1.0.1
> Environment: $ python --version
> Python 3.6.12
> $ pip list | grep pyarrow
> pyarrow 1.0.1
> Reporter: Alexander Dmitriev
> Priority: Minor
> Labels: pyarrow-serialization
> Attachments: pyarrow_recursion.py
>
>
> When serializing deeply nested python structures such as:
> {code:java}
> data =
> [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[None]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]{code}
> or
> {code:java}
> data = OrderedDict([('a', OrderedDict([('a', OrderedDict([('a',
> OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a',
> OrderedDict([('a', OrderedDict([('a', None)]))]))]))]))]))]))]))]))]){code}
> pyarrow raises the following exception:
> {code:java}
> >>> buffer = pyarrow.serialize(data).to_buffer()
> Traceback (most recent call last):
> File "pyarrow_recursion.py", line 11, in try_recursion
> buffer = pyarrow.serialize(data).to_buffer()
> File "pyarrow/serialization.pxi", line 285, in
> pyarrow.lib.SerializedPyObject.to_buffer
> File "pyarrow/serialization.pxi", line 248, in
> pyarrow.lib.SerializedPyObject.total_bytes.__get__
> File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Max recursion depth reached
> {code}
> While this not the prime use case of pyarrow.serialize, it would be nice to
> have a much larger recursion limit here, to not run into this issue with
> auto-generated structures quite so often.
> An example script for reproducing the error can be found in the attachments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
