potiuk commented on code in PR #30127:
URL: https://github.com/apache/airflow/pull/30127#discussion_r1139831681
##########
setup.py:
##########
@@ -399,6 +399,8 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT
/ "airflow" / "git_ve
"wheel",
"yamllint",
"aioresponses",
+ # TODO: why??
+ "aiobotocore>=2.1.1",
Review Comment:
One more thing here. Technically with the job implemented as it is right now
wiht my #30144 we have technically another possibility. We **could** reverse
the approach.
Currently we remove `aiobotocore` in the separate job and upgrade to latest
boto and run all the relevant provider tests.
But we could do it the opposite: we could use latest boto/botocore in
"main", remove aibotocore from the "devel" set of dependencies and install
aiobotocore (together with downgrading boto/botocore) in the separate job.
This has slightly different characteristics:
* by default the constraints we publish will not have aiobotocore and
willhave botocore/boto not compatible with the latest aiobotocore - this means
that if someone would like to use deferrable operators from AWS would have to
deliberately install aiobotocore and downgrade boto/botocore with it.
* the separate job would test automatically if the "aiobotocore" compatible
boto still works with all the unit-tested provider operators - this way any PR
that would be based on newer features would fail at PR time.
* then we could make a deliberate decision that this is ok (and add
conditional skips in the unit tests) if that happens
So basically this is this trade-off:
* (current) deferrable operators work out-of-the-box with the "official
constraints" (but without latest boto/botocore)
* (possible) - deferrable operators will not work with official constraints
and they require deliberately installing aiobotocore (and downgrading
boto/botocore) - but latest botocore is used in those constraints
Those are the two trade-offs, and we can still change the decision (easily)
which way go. With #30144 this is as easy as changing few lines in the CI
scripts.
##########
setup.py:
##########
@@ -399,6 +399,8 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT
/ "airflow" / "git_ve
"wheel",
"yamllint",
"aioresponses",
+ # TODO: why??
+ "aiobotocore>=2.1.1",
Review Comment:
One more thing here. Technically with the job implemented as it is right now
wiht my #30144 we have another possibility. We **could** reverse the approach.
Currently we remove `aiobotocore` in the separate job and upgrade to latest
boto and run all the relevant provider tests.
But we could do it the opposite: we could use latest boto/botocore in
"main", remove aibotocore from the "devel" set of dependencies and install
aiobotocore (together with downgrading boto/botocore) in the separate job.
This has slightly different characteristics:
* by default the constraints we publish will not have aiobotocore and
willhave botocore/boto not compatible with the latest aiobotocore - this means
that if someone would like to use deferrable operators from AWS would have to
deliberately install aiobotocore and downgrade boto/botocore with it.
* the separate job would test automatically if the "aiobotocore" compatible
boto still works with all the unit-tested provider operators - this way any PR
that would be based on newer features would fail at PR time.
* then we could make a deliberate decision that this is ok (and add
conditional skips in the unit tests) if that happens
So basically this is this trade-off:
* (current) deferrable operators work out-of-the-box with the "official
constraints" (but without latest boto/botocore)
* (possible) - deferrable operators will not work with official constraints
and they require deliberately installing aiobotocore (and downgrading
boto/botocore) - but latest botocore is used in those constraints
Those are the two trade-offs, and we can still change the decision (easily)
which way go. With #30144 this is as easy as changing few lines in the CI
scripts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
