o-nikolas commented on PR #73301:
URL: https://github.com/apache/airflow/pull/73301#issuecomment-5786643255
Myself and a few others have been playing with this new connection driven
approach and it works. But it's been a little confusing for folks and
explaining it to them has been tricky.
The questions that keep coming up are all variations of "so what do I
actually have to create and why?":
- Two connections are involved. One (type: `duckdb_aws`, default
`duckdb_aws_default`) says use AWS, the other (type: `aws`, default
`aws_default`) holds the actual credentials. Following the precedent of what
redshift and athena already do. **The difference is that a redshift or athena
connection actually describes something: a cluster host, a database, a region,
a workgroup. DuckDB runs in-process, so there's no endpoint to describe and the
connection is usually entirely empty**. So users are forced to create a blank
connection whose only purpose is to name a type to point DuckDB in the right
direction. Nothing creates it for them either and it's all a bit confusing.
- The two connections are also specified differently: the first by the
DuckDBExecuteQueryOperator's conn_id, the second by aws_conn_id (which can only
be set through hook_params).
None of that is impossible to figure out, but it has proven confusing so far.
So in the name of user experience (the original objection) what do folks
think about keeping the connection-driven SQL selection as it is now and also
adding the thin AwsDuckDBOperator (that packages up this connection work for
you, no empty connections needed to be created). It forces nobody to swap
operators, so Ash's original concern is met, but it gives people an easier
route if they just want basic functionality quickly? Both of them life side by
side just fine.
Let me know what you think @potiuk and @ashb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
