char101 commented on issue #39671: URL: https://github.com/apache/arrow/issues/39671#issuecomment-1902084793
Fastparquet also can append existing file by rewriting the footer https://github.com/dask/fastparquet/blob/fb545a5d8147eb111eded0d5ac11eda03c574134/fastparquet/writer.py#L966-L984 Unfortunately fastparquest can't do delta encoding yet and I find that with time series data, delta encoding can reduce the compression by 30% more. It would be great if this can be added to pyarrow too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
