potiuk opened a new pull request, #36591:
URL: https://github.com/apache/airflow/pull/36591
There was an edge case where parallel spell-check only could have failed
because of interaction between packages that were built in parallel.
Our documentation building is parallelized so that packages can be built in
parallel, however there is an extra complexity added when different packages
are built together in parallel and when those packages are referring to each
other.
The problem is when one package is using reference to another package that
is built at the same time. In this case the first package will fail, because it
will miss references to the other package.
This problem has been solved for documentation building, by introducing
multiple passess. If a package fails during the first wave, it is attempted to
be rebuilt for the second time (without rebuilding the packages that succeeded)
- this way at the second pass, there is no risk that those two pckages will be
built in parallel.
In fact we have up to 3(!) passes - in order to account for situation where
there are three packages in chain built together. I.e if there is A->B->C
dependency chain and A and B are built when C is being built both A and B will
fail in the first pass, but when A and B will be built in the second pass, A
will also fail (because it will expect B) so we have a third pass (and there
only A will be rebuilt).
This works fine for docs building, but when we do spellcheck-only, the
situation is a bit different because we do not perform documentation build in
the first place - so re-running spellchecking when we have some A-> B or A->B-C
dependency, because neither B nor C will be rebuilt when we run spellcheck-only.
This was attempted to be fixed in #36336 by switching to docs+spellcheck
build when we detect that spellchecking fails because of dependency to another
package. However the fix was not good enough - what the fix did was to attempt
to rebuild all packages and continue rebuilding all packages in 2nd and 3rd
pass - which had the unfortunate side-effect that if the 2nd pass failed as
well, the 3rd pass cleaned everything and repeated what the 2nd pass did before.
This PR fixes it properly:
* In case first pass with spellcheck-only fails because of package
dependency, the 2nd pass will rebuild all originally built packages both build
+ spellcheck
* However in this case the 3rd pass will only rebuild packages that failed
in the 2nd pass (not all originally built packages as it was implemented in
#36336)
* Also in case we do spellcheck-only, we introduce the 4th(!) pass to
account for the same case A->B-C that the third pass addresses in the "build
docs" case
This PR adds typing, reviews return values from the methods implementing the
logic and adds diagnostic logs and comments explaining the logic in an attempt
to make it more readable and see what's going on.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
