This is a simplified example but trying to figure out what gains can be had
using arrow vice straight nested python dictionaries for something like the
following:
{'random string 1': {'field1': {'field11': 'random string 2',
'field12': 100},
'field2': 200,
'field3': [300,
400,
{'random string 3': 500}]
},
'random string 4': {'field5': {'field51': 600,
'field52 ': [700,
800,
{'random string53': 900,
'random string54': 'random
string55'}
]
}
}
}
I didn't see anything that would convert an arbitrary nested dictionary
into some arrow structure -- did I miss something? If there isn't what are
some suggestions. I am doing pretty heavy data analysis where I am handed
some nested python dictionaries or nested json that I am loading into a
nested python dictionary. The memory footprint on these are large and I
have individual json files when loaded by json.load becomes a 5-6 GB python
dictionary (which is a little crazy when the actual json files is like
700MB).
curious,
Luke