ghaarsma commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5124337548
Thank you @raulcd and @rok for helping to investigate.
- I can confirm that we are using provided wheels from PyPi. We do not build
PyArrow ourself.
- I will try to work on a MRE. This will likely have to be pretty synthetic
and it might take me a while to extract the core behavior and try to reproduce.
- I have run pd.read_parquet timings with both pre_buffer settings on both
pyarrow 24 & 25. A total of 834 parquet files (~30 GB). Times are total seconds
to read all 834 files. Each timing is repeated 3 times.
```
pyarrow 24.0.0 --------------------------------------
1st 2nd 3rd
pd.read_parquet(prebuffer=False) 592.1 606.4 607.1
pd.read_parquet(prebuffer=True) 609.4 635.3 623.3
pyarrow 25.0.0 --------------------------------------
1st 2nd 3rd
pd.read_parquet(prebuffer=False) 634.2 623.4 628.6
pd.read_parquet(prebuffer=True) 663.5 641.8 665.2
```
- Pandas version is 3.0.5 and has not changed. We use reproducible builds
with a lock file. All DateTimeIndex are timezone-aware and all should be UTC. I
have checked all files and here is where it gets perhaps a bit interesting:
```
pyarrow 24.0.0 reports: <class 'datetime.timezone'>, 'datetime.timezone.utc'
pyarrow 25.0.0 reports: <class 'zoneinfo.ZoneInfo'>,
"zoneinfo.ZoneInfo(key='UTC')"
```
- The query length (s) on the figure is the time in seconds to run a
`warming the cache` loop. Cache hit/miss behavior should not impact the results
more then a few %.
Hoping this helps for know. Could it be that the timezone awareness
difference between pyarrow 24/25 as datetime.utc and ZoneInfo("UTC") be the
cause? What if we have to stich `pd.concat` timeseries data together all based
on UTC but differ between datetime and ZoneInfo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
