jorisvandenbossche commented on issue #50667: URL: https://github.com/apache/arrow/issues/50667#issuecomment-5219558937
Thanks for the further research, that is interesting! > BTW I could not find the change in the [release notes](https://arrow.apache.org/release/25.0.0.html) The actual change was in https://github.com/apache/arrow/pull/49694 (I think it didn't how up properly in the release notes because I did not mark / close the issue for the 25.0 milestone ..) > What I still don't understand is that out of my 831 cached TimeSeries / parquet files, in pyarrow 25.0.0, when concatenating with a second datetime.UTC indexed dataframe, 56 will result in a ZoneInfo("UTC"), without any slowdown. The other 775 will result in a datetime.UTC index and this is where all the slowdown happens. That is indeed strange .. The logic should be (in pyarrow 25) to first try to convert the tz string to `zoneinfo.ZoneInfo`, and if that does not succeed, fallback with trying to convert it to `pytz`. In pyarrow 24, it just always first converted to `pytz`, and only if that was not available, try `zoneinfo`. But I don't fully understand when you would end up with a `datetime.UTC` timezone. Something that might be useful to check: could you verify in the Parquet files which exact timezone is being used? If you read the data with pyarrow and check for the schema of the resulting table, i.e. `pyarrow.parquet.read_table(..).schema`, which timezone does it show for the timeseries column? (e.g. I am wondering if capitalization could trigger some different behaviour) --- Regardless of figuring out how you get the different timezones, I am also not actually seeing a slowdown in `pd.concat` in this case: ```python >>> N = 10_000_000 >>> import zoneinfo >>> dates = pd.date_range("2020-01-01", periods=N, freq="min", tz="UTC") ... df = pd.DataFrame({"col": np.random.randn(N)}, index=dates) >>> df.index.dtype.tz datetime.timezone.utc >>> dates = pd.date_range("2020-01-01", periods=N, freq="min", tz=zoneinfo.ZoneInfo("UTC")) ... df2 = pd.DataFrame({"col": np.random.randn(N)}, index=dates) >>> df2.index.dtype.tz zoneinfo.ZoneInfo(key='UTC') >>> df_copy = df.copy() # concatting two dataframes with `datetime.timezone.utc` >>> %timeit pd.concat([df, df_copy]) 77.8 ms ± 3.64 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) # concatting dataframe with `datetime.timezone.utc` and `zoneinfo.ZoneInfo(key='UTC')` >>> %timeit pd.concat([df, df2]) 79.5 ms ± 4.54 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
