Hi everyone ! I was wondering: What is the most efficient way to know the number of rows in dataset of Arrow IPC files ?
I expected each file to have the number of rows as metadata in the footer, but it doesn’t seem to be the case. Therefore I need to call count_rows() which is less efficient than reading metadata. Maybe the number of row can be written as custom_metadata in the footer, but the writing/reading custom_metadata functions don’t seem to be exposed in python - if I’m not mistaken. Thanks in advance :) -- Quentin
