potiuk commented on a change in pull request #6733: [AIRFLOW-6058] Fixed
remnant of nose tests (fixup)
URL: https://github.com/apache/airflow/pull/6733#discussion_r354259363
##########
File path: tests/operators/test_hive_operator.py
##########
@@ -156,9 +155,6 @@ def test_mapred_job_name(self, mock_get_hook):
if 'AIRFLOW_RUNALL_TESTS' in os.environ:
Review comment:
No. It's not this. The AIRFLOW_RUNALL_TESTS is not set anyway (and never was
in CI). It is something that we should handle separately as part of AIP-4
(support for system tests) - this is a way by someone to skip running system
tests on a CI build but have a way to run them locally (but it's not
consistent/standardized as AIP-4 is supposed to do).
I tracked it down and It is a bug in the Dockerfile in the way how
dependencies are treated in the CI optimised build. I will create a JIRA ticket
for that in a moment.
It was supposed to be handled by this:
```
# Increase the value here to force reinstalling Apache Airflow pip
dependencies
ARG PIP_DEPENDENCIES_EPOCH_NUMBER="2"
ENV PIP_DEPENDENCIES_EPOCH_NUMBER=${PIP_DEPENDENCIES_EPOCH_NUMBER}
```
This should trigger reinstallation of all dependencies from the scratch in
case we know we deleted some dependencies (nose in this case). This is done via:
```
pip install \
"https://github.com/apache/airflow/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]";
\
&& pip uninstall --yes apache-airflow;
```
However in case of PR from another repo, AIRFLOW_BRANCH is still set to
"master" (this is the TARGET branch) and it uses "airflow" Github
organisation/user.
This means that in the PR, dependencies will be installed first from the
master branch of Airflow. Which (at the time of PR) included nose.
What it SHOULD happen instead - in case of PR, it should be something like:
```
pip install \
"https://github.com/apache/${SOURCE_GITHUB_USER}/archive/${SOURCE_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]";
\
&& pip uninstall --yes apache-airflow;
```
This perfectly explains what happened.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
