This is a simplified example but trying to figure out what gains can be had
using arrow vice straight nested python dictionaries for something like the
following:

{'random string 1': {'field1': {'field11': 'random string 2',
                                'field12': 100},
                     'field2': 200,
                     'field3': [300,
                                400,
                                {'random string 3': 500}]
                    },
 'random string 4': {'field5': {'field51': 600,
                                'field52 ': [700,
                                            800,
                                            {'random string53': 900,
                                             'random string54': 'random
string55'}
                                            ]
                                 }
                     }
}

I didn't see anything that would convert an arbitrary nested dictionary
into some arrow structure -- did I miss something?  If there isn't what are
some suggestions.  I am doing pretty heavy data analysis where I am handed
some nested python dictionaries or nested json that I am loading into a
nested python dictionary.  The memory footprint on these are large and I
have individual json files when loaded by json.load becomes a 5-6 GB python
dictionary (which is a little crazy when the actual json files is like
700MB).

curious,
Luke

Reply via email to