potiuk commented on pull request #8991: URL: https://github.com/apache/airflow/pull/8991#issuecomment-633248384
@ashb @kaxil @BasPH @mik-laj I think I nailed it with import tests for backport packages. I removed all the README changes from code changes so that it will be easier to review that one. I will regenerate the READMES again in the separate commit on top of that one. I implemented all the refactorings that were needed in automated way using Bowler. I actually started to like Bowler :). I got to the point however that I had to refactor it - we had one huge method that did everything - so I separated all the refactoring code out from setup_backport_packages and I converted it into a separate class (this way Query was easier to build as a field in the class rather than passing the query around. @turbaszek -> you might want to take a look and see if you are ok with that, but I think the way it is done now is much more readable - we have separate refactoring methods for separate classes. Now - we also have two separate scripts for testing packages - one is to install them one-by-one for airflow 1.10 and see if they can be installed and the second one installs all of them, finds all the relevant classes and imports them (and fails if some of those imports fail). It's done rather neatly and for us It is I think a great tool to keep backport packages usable in the future - as those tests will be running continuously in CI with every PR. @BasPH -> it was a really good idea. I see some of the changes coming (like functional DAGs) might unknowingly destroy the backward compatibility of provider packages (similar to lineage changes for Papermill) - but with this test, it will be a lot harder to introduce such problems. One last comment for Papermill - it's AUTO lineage feature caused import of examples to fail (yes - we are also checking the examples!), So I added info that AUTO lineage will not work in 1.10 (and I updated the example so that it behaves correctly in Airflow 1.10 as well) @bolkedebruin - you might want to take a look at this part to see if I did not introduce any problems. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
