jorisvandenbossche commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5207567806
I tried to reproduce based on your description of a DataFrame with a single
float column and a timeseries index:
```python
import time
import numpy as np
import pandas as pd
N = 10_000_000
dates = pd.date_range("2020-01-01", periods=N, freq="h", tz="UTC")
df = pd.DataFrame({"col": np.random.randn(N)}, index=dates)
write_times = []
read_times = []
for _ in range(10):
t0 = time.time()
df.to_parquet("test.parquet")
t1 = time.time()
df2 = pd.read_parquet("test.parquet")
t2 = time.time()
write_times.append(t1 - t0)
read_times.append(t2 - t1)
print("Writing took :", sum(write_times))
print("Reading took :", sum(read_times))
```
and compared pyarrow 24 to 25 (but using conda packages, not pypi packages),
and don't see any significant difference:
```
$ pixi exec --with pyarrow=24 --with pandas=3.0.5 python
test_pyarrow_parquet_perf.py
Writing took : 5.342243432998657
Reading took : 1.740952730178833
$ pixi exec --with pyarrow=25 --with pandas=3.0.5 python
test_pyarrow_parquet_perf.py
Writing took : 5.0998406410217285
Reading took : 1.74660325050354
```
@ghaarsma could you run this code on your end to see if that reproduces the
slowdown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
