I am reading in a parquet file and I want to loop over each of the rows and
right now I am converting to a pandas dataframe and then using
pandas.DataFrame.itertuples to access each row. Is there a way to do this
all in pyarrow and not convert to pandas? Just looking at ways to
optimize.
One thing I have tried is to convert the pyarrow table to a list of
dictionaries that are then looped over, but that is much slower in my case
than the conversion to pandas and using itertuples. I was surprised it was
slower.
What I mean by that is given pyarrow table t (read from
pyarrow.parquet.read_table)
new_list = []
for i in range(t.num_rows):
new_dict = dict()
for k in t.column_names:
new_dict[k] = t[k][i].as_py()
new_list.append(new_dict)
now loop over new_list.
The method pyarrow.Table.to_pydict() would be helpful to make the above
more concise but I need it oriented like pandas.DataFrame.
to_dict('records').
I get this might not be implemented yet, just asking in case I am missing
how to do this natively in arrow.
Thanks,
Luke
