dabla commented on code in PR #74691: URL: https://github.com/apache/airflow/pull/74691#discussion_r4287570332
########## providers/amazon/docs/transfer/sql_to_s3.rst: ########## @@ -24,6 +24,10 @@ Use ``SqlToS3Operator`` to copy data from a SQL server to an Amazon Simple Stora converts the SQL result to `pandas dataframe <https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html>`__ (e.g. MySQL, Hive, ...). +The operator needs the ``pandas`` extra. Writing Parquet files (``file_format="parquet"``) or reading Review Comment: **[nit]** This only holds for the default `df_type="pandas"`. The operator also takes `df_type="polars"`, which needs polars, not the `pandas` extra, so the sentence reads as if pandas were always required. Scoping it to the default keeps it true. Something like: ```rst With the default ``df_type="pandas"`` the operator needs the ``pandas`` extra. Writing Parquet files (``file_format="parquet"``) or reading with ``read_kwargs={"dtype_backend": "pyarrow"}`` also needs pyarrow, which the ``parquet`` extra installs: ``apache-airflow-providers-amazon[pandas,parquet]``. ``` ("needs pyarrow" rather than "needs the extra", since pandas also accepts `fastparquet` as the Parquet engine.) ########## providers/amazon/pyproject.toml: ########## @@ -136,6 +136,11 @@ dependencies = [ 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', 'pandas>=2.3.3; python_version >="3.14"', ] +"parquet" = [ Review Comment: **[nit]** Optional: the runtime error still doesn't point at the new extra. A user without pyarrow still gets pandas' "Unable to find a usable engine" from `SqlToS3Operator.execute`, which says nothing about `apache-airflow-providers-amazon[parquet]`. The operator already raises `AirflowOptionalProviderFeatureException` for a missing pandas in `_fix_dtypes`; the same for the Parquet engine (pandas path only, and accepting `fastparquet` too, so it doesn't break users of that engine) would make the extra discoverable. Fine as a follow-up; the exact line is in `transfers/sql_to_s3.py`, outside the diff. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
