hi Suhail, This follows the columnar format closely. The List layout is composed from a child array providing the "inner" values, which are given the List<T> interpretation by adding an offsets buffer, and a validity buffer to distinguish null from 0-length list values. So flatten() here just returns the child array, which has only 3 values in the example you gave.
A function could be written to insert "null" for List values that are null, but someone would have to write it and give it a name =) - Wes On Wed, Sep 25, 2019 at 5:15 PM Suhail Razzak <suhail.raz...@gmail.com> wrote: > > Hi, > > I'm working through a certain use case where I'm unnesting ListArrays, but > I noticed something peculiar - null ListValues are not retained in the > unnested array. > > E.g. > In [0]: arr = pa.array([[0, 1], [0], None, None]) > In [1]: arr.flatten() > Out [1]: [0, 1, 0] > > While I would have expected [0, 1, 0, null, null]. > > I should note that this works if the None is encapsulated in a list. So I'm > guessing this is expected logic and if so, what's the reasoning for that? > > Thanks, > Suhail
