Taragolis commented on PR #33602:
URL: https://github.com/apache/airflow/pull/33602#issuecomment-1697069722
I'm still not sure that hook should be a part which are responsible for
connection, but we have what we have. If go thought community providers we
easily could find than many of them implements it's own unique method with
Connection integration and resolving merging parameters.
Let's me show how I would some simple integration, where parameters for
integration do not lost between operator and hook, pretty well documented,
could be validated before use (now we validate only in the UI) and also could
use as part of build UI and auto-documentation.
```python
class AwesomeConnection(BaseModel):
"""Awesome Connection"""
class Config:
...
connection_id: str = Field(
title="Name",
description="The name value of insight",
)
login: str = Field(
title="Awesome Login",
description="Description to Awesome ID",
airflow_connection_only=True,
)
password: str = Field(
title="Awesome Password",
description=(
"Description to Awesome Password, e.g. "
"where it could be obtained in case cloud services"
),
airflow_sensetive=True,
airflow_connection_only=True,
)
hostname: str = Field(
title="Awesome Hostname",
description="Foo bar spam egg",
)
code_only_parameter: str = Field(
title="Awesome Parameter",
description="Foo bar spam egg",
airflow_user_code_only=True,
)
class AwesomeHook:
def __init__(self, *, connection: AwesomeConnection):
self._connection = connection
@cached_property
def client(self):
resolved_connection_info = awesome_common_merger(self._connection)
# or
resolved_connection_info =
self._connection.merge_with(conn_id=self._connection.connection_id)
return awesome_client(
login=resolved_connection_info.login,
password=resolved_connection_info.password,
hostname=resolved_connection_info.hostname,
code_only_parameter=resolved_connection_info.hostname,
)
def do_something_awesome(self, param1, param2):
return self.client.sudo_rm_rf_root_dir(param1, param2,
preserve_root=False)
def do_something(self, param1, param3):
return self.client.sleep(param1, param3)
class AwesomeOperatorDoSomethingAwesome(BaseOperator):
def __init__(self, *, connection: AwesomeConnection, param1, param2,
**kwargs):
super().__init__(**kwargs)
self._connection = connection
self.param1 = param1
self.param2 = param2
@cached_property
def hook(self):
return AwesomeHook(connection=self._connection)
def execute(self, context):
return self.hook.do_something_awesome(self.param1, self.param2)
class AwesomeOperatorDoSomething(BaseOperator):
def __init__(self, *, connection: AwesomeConnection, param1, param3,
**kwargs):
super().__init__(**kwargs)
self._connection = connection
self.param1 = param1
self.param3 = param3
@cached_property
def hook(self):
return AwesomeHook(connection=self._connection)
def execute(self, context):
return self.hook.do_something(self.param1, self.param3)
```
This primitive solution wouldn't work well with something abstracted like
HttpHook.
And I can't find the way how to start moving from "put whatever you wanted
to connection, hook and you could even not extend the documentation" to some
partially-strict, which do not give a chance to use something unsafe from
connection, and restrict provide sensitive information directly thought user
code (tokens/passwords and etc.)
Maybe there is some intermediate solution exists without break everything
for everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
