Hi Max, Did you take a look at Poetry (https://poetry.eustace.io/ <https://poetry.eustace.io/>)? How does this PR compare?
-ash > On 24 Jun 2019, at 22:02, Maxime Beauchemin <[email protected]> > wrote: > > This Superset PR may be relevant and useful for Airflow too (still WiP): > https://github.com/apache/incubator-superset/pull/7762 > > Basically it allows your project to define a "requirements/" folder with a > "requirements.json" as a single source of truth for all deps, including the > details of what goes in the "extra_requires", and compiles/pins all sets of > dependencies in a set of useful files > > Now: > * this requirements.json is parsed and used directly as is in `setup.py` > (same as before) > * running "compile_requirements.py" effectively creates a set of > "pip-compiled" full tree and pinned requirements for each set of > requirements. > * these pinned files can be used in CI / docker / builds to avoid bad > surprises like it appears you did. The package (setup.py) is still > permissive and allows people to pick their versions, but CI can be pinned > and rock solid. > > The folder includes something like: > requirements/requirements-hive.in > requirements/requirements-hive.txt > requirements/requirements-postgres.in > requirements/requirements-postgres.txt > # ... (2 files for each extra_requires defined in the JSON file) > > There may be some subtleties around py2/py3 reqs diverging, but you could > probably bake that into that logic if you wanted to (for Superset it's a > non-issue since we only support py36). > > Max > > On Mon, Jun 24, 2019 at 1:16 AM Kaxil Naik <[email protected]> wrote: > >> It works :) - The CI passed for the commit on v1-10-test, so your PR should >> pass as well. >> >> On Mon, Jun 24, 2019 at 1:40 PM Jarek Potiuk <[email protected]> >> wrote: >> >>> I pinned auto-api to below 1.1.0 and re-run the test - Let's hope it >> works >>> for now . >>> >>> On Mon, Jun 24, 2019 at 10:05 AM Jarek Potiuk <[email protected]> >>> wrote: >>> >>>> OK. After some checks it turns out it's not only Sphinx really but also >>>> sphinx-autoapi (v 1.1.0 released yesterday). >>>> >>>> J. >>>> >>>> >>>> On Mon, Jun 24, 2019 at 9:29 AM Jarek Potiuk <[email protected] >>> >>>> wrote: >>>> >>>>> Kamil prepared a fix https://github.com/apache/airflow/pull/5468 >>> (thanks!) and >>>>> when I tested it on a clean pip-install with latest sphinx it seems to >>> work >>>>> fine (Running Travis CI build >>>>> <https://travis-ci.org/apache/airflow/builds/549600100>now). Let's >> see >>>>> if it helps. >>>>> >>>>> BTW. The AIP-10 Docker image last step (pending Kubernetes fix) should >>>>> solve most of the dependency issues, however I think this particular >>>>> problem could be missed (it's an interesting one). >>>>> >>>>> I will soon start another thread (after dockerfile merging) about >>> finally >>>>> looking into making dependencies more manageable and less >> transient-deps >>>>> problems. I already have some ideas how we can approach the problem >> that >>>>> Airflow is both: library (no-dependency pinning recommended) and >>>>> end-product (strong pinning recommended). >>>>> >>>>> J. >>>>> >>>>> On Mon, Jun 24, 2019 at 8:48 AM Ash Berlin-Taylor <[email protected]> >>> wrote: >>>>> >>>>>> Any cyclic imports we have aren't actually a problem and are handled >>>>>> fine by python 3 (obviously, else it wouldn't work) so I think this >> is >>> a >>>>>> big in sphinx with it being over zealous. >>>>>> >>>>>> This only affects tests and development, not our users in this case. >>>>>> >>>>>> If it's difficult to fix I would go with pinning - I want to get a >>>>>> 1.10.4 beta 1 out today. >>>>>> >>>>>> -ash >>>>>> >>>>>> On 24 June 2019 00:37:57 BST, Jarek Potiuk <[email protected] >>> >>>>>> wrote: >>>>>>> >>>>>>> Hello Everyone, >>>>>>> >>>>>>> We have yet another case where relesing a package (Sphinx 2.1.2 in >>> this >>>>>>> case) - started to cause master to fail. >>>>>>> >>>>>>> Should we pin Sphinx to an earlier version or maybe someone can >> help >>> and >>>>>>> figure out and fix it quickly (seems that there are some cyclic >>> imports >>>>>>> that probably need solving anyway if Sphinx is right about them) ? >>> @Kamil >>>>>>> BreguĊa <[email protected]> @Fokko Driesprong >>> <[email protected]> >>>>>>> ? WDYT ? >>>>>>> >>>>>>> I created an issue for that >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4841 >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Jarek Potiuk >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>> >>>> M: +48 660 796 129 <+48660796129> >>>> [image: Polidea] <https://www.polidea.com/> >>>> >>>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >> >> >> -- >> *Kaxil Naik* >> *Big Data Consultant | DevOps Data Engineer* >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j >> Developer >> *LinkedIn*: https://www.linkedin.com/in/kaxil >>
