mobuchowski commented on PR #42963: URL: https://github.com/apache/airflow/pull/42963#issuecomment-2410860673
>The root cause seem to be that openlineage uses "redshift_connector" as sqlalchemy dialect internally - and while direct import is not present anywhere, it expects the library to be present as it contributes the "redshift_connector" dialect. >This is why we think the dependency is not used, but in fact it is. Exactly, we use it to generate queries fitting the right db. We've migrated to this approach from [earlier approach when we manually slapped SQL together via string concatenation.](https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/dbapi_utils.py#L111) IMO what could work is [moving Redshift hook to use Postgres dialect](https://github.com/apache/airflow/blob/857ca4c06c9008593674cabdd28d3c30e3e7f97b/providers/src/airflow/providers/amazon/aws/hooks/redshift_sql.py#L166) - unless there is something breaking using Postgres dialect from working with Redshift now; it was used for this purpose before actual Redshift dialect. BTW, IMO the fact that there are no tests other than OL breaking from this means that the `get_sqlalchemy_engine` for Redshift isn't tested, not that only OL uses Redshift dialect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
