ghaarsma commented on issue #50667: URL: https://github.com/apache/arrow/issues/50667#issuecomment-5218934890
I have not yet been able to generate a fully syntactic example (that I can share) to demonstrate the slowdown. But I have made progress in determining where the slowdown is. I have also reproduced the problem on Windows (our local Dev env). The actual slowdown is not in the pandas read_parquet or to_parquet (via engine="pyarrow") as can be seen here: <img width="2417" height="1328" alt="Image" src="https://github.com/user-attachments/assets/471f2e64-60f9-4e55-a071-2548e7d6b424"; /> All the slowdown is caused in the pandas concat which concatenates the old and new dataframe together before writing it back out <img width="2420" height="1242" alt="Image" src="https://github.com/user-attachments/assets/ef59340d-b120-43ce-9e01-d07c35d50b96"; /> I can confirm that the slowdown goes away if both dataframes have identical DatetimeIndex timezones. Either both datetime.UTC or both ZoneInfo("UTC"). So I think it would be fair to say that the bug is more in Pandas, but that it has been induced by the Pyarrow 25.0.0 change to use ZoneInfo("UTC") instead of datetime.UTC. The slowdown is quite significant, about ~ 365 times slower, when comparing the slopes of the scatter plots in the 2nd figure. - Would it be possible to go back to datetime.UTC as was in pyarrow 24.0.0? BTW I could not find the change in the [release notes](https://arrow.apache.org/release/25.0.0.html) - What I still don't understand is that out of my 831 cached TimeSeries / parquet files, in pyarrow 25.0.0, when concatenating with a second datetime.UTC indexed dataframe, 56 will result in a ZoneInfo("UTC"), without any slowdown. The other 775 will result in a datetime.UTC index and this is where all the slowdown happens. I'll continue to investigate the 2nd point, because I think this is the key to a reproducible example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
