[jira] [Created] (ARROW-9994) [C++][Python] Auto chunking nested array containing binary-like fields result malformed output

[Krisztian Szucs (Jira)]

Krisztian Szucs created ARROW-9994:
--------------------------------------

             Summary: [C++][Python] Auto chunking nested array containing 
binary-like fields result malformed output
                 Key: ARROW-9994
                 URL: https://issues.apache.org/jira/browse/ARROW-9994
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++, Python
    Affects Versions: 1.0.0
            Reporter: Krisztian Szucs
            Assignee: Krisztian Szucs


In case of nested types the binary-like arrays are chunked but not the others, 
so after finalizing the builder the nested output array contains different 
length children.

{code:python}
   char = b'x'
   ty = pa.binary()

    v1 = char * 100000000
    v2 = char * 147483646

    struct_type = pa.struct([
        pa.field('bool', pa.bool_()),
        pa.field('integer', pa.int64()),
        pa.field('string-like', ty),
    ])

    data = [{'bool': True, 'integer': 1, 'string-like': v1}] * 20
    data.append({'bool': True, 'integer': 1, 'string-like': v2})
    arr = pa.array(data, type=struct_type)
    assert isinstance(arr, pa.Array)

    data.append({'bool': True, 'integer': 1, 'string-like': char})
    arr = pa.array(data, type=struct_type)
    assert isinstance(arr, pa.ChunkedArray)
{code}

{code:python}
len(arr.field(0)) == 22
len(arr.field(1)) == 22
len(arr.field(2)) == 1  # the string array gets chunked whereas the rest of the 
fields do not
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)