potiuk commented on a change in pull request #4353: [AIRFLOW-3480] Add Database
Deploy/Update/Delete operators
URL: https://github.com/apache/incubator-airflow/pull/4353#discussion_r244038274
##########
File path: airflow/contrib/hooks/gcp_spanner_hook.py
##########
@@ -147,15 +149,13 @@ def _apply_to_instance(self, project_id, instance_id,
configuration_name, node_c
:param func: Method of the instance to be called.
:type func: Callable
"""
- client = self.get_client(project_id)
- instance = client.instance(instance_id,
- configuration_name=configuration_name,
- node_count=node_count,
- display_name=display_name)
+ instance = self.get_client(project_id).instance(
Review comment:
I'd rather wait with hook tests to subsequent PRs (and maybe we solve it in
a bit different way). I think it is rather difficult to test automatically the
hooks to GCP using mocking/classic unit tests of Airflow. Please let me know
what you think about the approach we use? Maybe we are missing something and we
can add some unit test for hooks which could add value :)?
We chose a slightly different strategy for testing GCP operators (and hooks):
* First of all we try to make hooks as straightforward as possible -
basically 1-1 operation of an existing library method but we make it
synchronous - waiting for operation to complete. Then we put pretty much all
logic into operators.
* What we realised is that we can mostly test with mocking and unit tests in
this case, is whether this particular library method has been called. Which is
a bit redundant - that's why we do not have those hook tests.
* instead we run automated system tests ("we" means the team from my company
Polidea - 3 people who work on those operators). We use example DAGs
(`example_gcp_spanner.py` in this case) to run them automatically with a real
GCP_PROJECT. We even have a way to run them using (skipped by default) unit
testss (see for example CloudSpannerExampleDagsTest). We have a separate and
quite sophisticated environment for that - we run it automatically in GCP Cloud
Build on every push and this way we test e-2-e whether the operators (and thus
hooks) work with real GCP.
I am going to try to contribute it very soon to the community (it's already
open-sourced and I am polishing it) so that others will be able to do the same
with their own GCP projects. You can see it here
https://github.com/PolideaInternal/airflow-breeze and I will be happy to
involve you for comments/re view when we are ready to share it (I think in a
couple of days).
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
