[
https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-2871:
--------------------------------
Attachment: screenshot-2.png
> Harden and improve Read the Docs build environment
> --------------------------------------------------
>
> Key: AIRFLOW-2871
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
> Project: Apache Airflow
> Issue Type: Bug
> Components: docs, Documentation
> Reporter: Taylor Edmiston
> Assignee: Taylor Edmiston
> Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR
> 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some
> oddities in our Read the Docs (RTD) build environment especially around
> cached dependencies. This motivates hardening and showing some love to our
> RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock
> dependency discussed in PR #3703 above. I think that our RTD environment
> possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip
> install --ignore-installed --cache-dir
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip
> .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens
> to have mock installed already.
> {code:java}
> python
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip
> install --upgrade --cache-dir
> /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip
> Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1
> alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8
> sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
> Here are some logs where you can see that (view raw):
> # Latest successful (Aug. 7, 2018. 9:21 a.m.) -
> [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
> # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) -
> [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
> # Last successful before (2) (July 18, 2018. 3:23 a.m.) -
> [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
> # First build (2016) -
> [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first
> RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in
> our current config in incubator-airflow; I believe they are installed as
> [core dependencies of RTD
> itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further
> down in the build. In the case of mock, we were getting lucky that
> mock==1.0.1 was a dependency of RTD and our setup inherited that old version
> which allowed the docs build to succeed. (There might be other cases of
> dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
> * Hardening
> ** Set our RTD build to use a virtual environment if it's not already
> ** Set our RTD build to ignore packages outside of its virtualenv like
> dependencies of RTD itself
> ** Specify any dependencies broken by ^
> ** Test wiping a version in the build environment (not sure if this clears
> cache dir)
> ***
> [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
> ***
> [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
> ** Make build.image, python.version, etc explicit in yaml config
> *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
> ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
> * Misc
> ** Improve RTD project page to have tags and description
> ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe
> this would require. I am happy to take this issue if I can get that. I have
> experience as an admin of another project on RTD (simple-salesforce).
> ----
> /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
