Dev-iL commented on code in PR #74817:
URL: https://github.com/apache/airflow/pull/74817#discussion_r4307061938
##########
providers/amazon/src/airflow/providers/amazon/aws/transfers/sql_to_s3.py:
##########
@@ -220,7 +221,23 @@ def _strip_suffixes(
return path[: -len(suffix)]
return path
+ def _ensure_parquet_engine(self) -> None:
+ needs_parquet = self.file_format == FILE_FORMAT.PARQUET or (
+ ("dtype_backend", "pyarrow") in self.read_kwargs.items()
+ )
+ if self.df_type != "pandas" or not needs_parquet:
+ return
+ if any(importlib.util.find_spec(engine) for engine in ("pyarrow",
"fastparquet")):
Review Comment:
Confirmed at the source rather than from the repro alone: in pandas 3.0.6,
`pandas/io/sql.py:174-175` does `if dtype_backend == "pyarrow": pa =
import_optional_dependency("pyarrow")`, so fastparquet cannot stand in for it
whatever the output format is.
Split the check: `dtype_backend="pyarrow"` now requires pyarrow
specifically, and only Parquet *output* accepts either engine. Two cases cover
it — `pandas-pyarrow-backend-fastparquet-only` and
`pandas-parquet-pyarrow-backend-fastparquet-only`, the second being the overlap
you flagged, where the output is Parquet and pyarrow is still the only thing
that will do. I checked both fail against the previous helper body (`DID NOT
RAISE`), so they are load-bearing.
The message now names the setting that triggered it, e.g.
`read_kwargs={"dtype_backend": "pyarrow"} needs pyarrow. Install
apache-airflow-providers-amazon[parquet].` The `[parquet]` extra resolves to
pyarrow only, so it stays the right pointer for both branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
