areusch opened a new pull request #6620:
URL: https://github.com/apache/incubator-tvm/pull/6620
Still a WIP, posting for people to take a look at. This should maybe become
an RFC, happy to write one up. I just needed a `pyproject.toml` for a project
I'm working on now so I'm posting it.
It's really hard to reproducibly use TVM outside the docker containers, and
depending on what hardware you have at work/home (are they different now?), you
can't always run the docker container you need to (I.e. ci-gpu). Clearly
documenting our python dependencies will help.
I think *just* checking this in as-is would be bad because:
- I'm just copying dependencies from one place to another. There isn't
anything to enforce that our docker images, and therefore our CI, actually uses
the selected versions
- That aside, it would be great to generate a constraints file or some
other log *outside* the docker images, so that you can have a clear record of
e.g. which version of `pylint` is used to lint without needing to pull the
container.
- In order to actually use `poetry`, I had to hack `setup.py` to hardcode
the library location. Maybe we should just do this. Right now if you have
`libtvm_runtime.so` in `PATH` then it just uses that one--very bad in
development! The reason for this is that dependency tracking tools installing
packages in editable mode really want to run something like `python setup.py
egg_info` to determine the library version, and unless you've built tvm, poetry
will just crash trying to do anything. that's a bad developer experience.
- Making `pyproject.toml` the authoritative source of dependencies (I.e.
rewriting the docker/install scripts to install only using `poetry`) would be
bad because non-users of poetry would have to transcribe the dependencies into
e.g. `requirements.txt`. Yes, tools exist to do this; we should do it for them.
Here are some future things I'd like to do to address these issues:
- [ ] Write a script that uses e.g. dephell to auto-generate
`python/requirements.txt` and `python/requirements-<extra>.txt` for each extra
section included.
- [ ] Write a unit test that asserts that `python/requirements*.txt` is in
sync with `pyproject.toml`.
- [ ] Make it easy to install all extras in one command (likely needs output
from previous script)
- [ ] Rewrite `docker/install/ubuntu_install_<extra>.sh` to just run `poetry
install -E <extra>`.
- [ ] Add `poetry lock` to the docker build process and export the generated
lockfile somehow...
- [ ] Decide what to do about `setup.py` library location hack.
Here's why I think we should use something other than requirements.txt as
the authoritative source of dependencies:
- In order to ensure a smooth user experience, we do have to pin our
dependencies to *some* degree. Here, I've pinned to the package minor or major
version where that made sense.
- However, of course pinning is bad unless you update your pinned deps. so
some dependency management tool provides that capability. I'm not at all
attached to poetry as the tool to use for that, but it can facilitate it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
