On Sat, 1 Sep 2018 at 11:02, Tzu-ping Chung <uranu...@gmail.com> wrote: > > I’m not knowledgable about GPUs, but from limited conversations with others, > it is important to first decide what exactly the problem area is. Unlike > currently > available environment markers, there’s currently not a very reliable way to > programmatically determine even if there is a GPU, let alone what that GPU can > actually do (not every GPU can be used by Tensorflow, for example).
As Tzu-Ping notes, using environment markers currently requires that there be a well-defined "Python equivalent" to explain how installers should calculate the install-time value of the environment marker. However, even regular CPU detection has problems when it comes to environment markers, since platform_machine reports x86_64 on a 64-bit CPU, even if the current interpreter is built as a 32-bit binary, and there are other oddities like Linux having two different 32-bit ABIs (there's i686, which is the original 32 bit ABI that predates x86_64, and then there's x32, which is the full x86_64 instruction set, but using 32-bit pointers: https://github.com/pypa/pip/issues/4962 ). (Also see https://github.com/pypa/pipenv/issues/2397 for some additional discussion) Given the complexity of the problem, what we may want to do is to go with a manylinux style solution, where even though installers are expected to make a minimal effort to figure out an answer on their own, a particular answer can also be forced by installing a module into the current environment that has a particular attribute set to True or False. (See https://www.python.org/dev/peps/pep-0571/#platform-detection-for-installers for details) For example, let's suppose we call the magic module "__installmarkers__": - dunder-name to indicate that it's a special metadata module, rather than a regular one - "install markers" rather than "environment markers", since they're not for general purpose information about the environment, they're specifically the markers that relate to declarations of installation dependencies Given that, the environment marker lookup rules could be amended to say to check for "__installmarkers__.<name>" before checking the regular definition, and otherwise hard-to-define cases like "Is a GPU available?" could be handled by: - expanding the environment marker syntax to allow for standalone flag attributes (i.e. no comparison operation, just a flag name) - expanding the environment marker syntax to allow for negation (i.e. a preceding unary "not") - define "have_gpu" as a new flag attribute that's assumed to be "False" by default - by default, conditional dependencies like "cpu-only-version; not have_gpu" will get installed - setting "__installmarkers__.have_gpu" to True will mean that conditional dependencies like "gpu-optimised-version; have_gpu" will get installed To help improve forwards compatibility in the future, it may even make sense to say that all installers should treat unknown names in environment markers as "getattr(__installmarkers__, NAME, None)", and all environment markers that they can't parse as False. Note that I don't think it's possible for folks to get away from the "3 projects" requirement if publishers want their users to be able to selectively *install* the GPU optimised version - when you're keeping everything within one project, then you don't need an environment marker at all, you just decide at import time which version you're actually going to import. Cheers, Nick. P.S. As an alternative to a magic module, the install marker overrides could be placed in pyvenv.cfg. Even if we did that, we'd probably still want the magic module option though, as pyvenv.cfg doesn't exist for user-level and interpreter-installation-level installs. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3DP67I7HDJXRRNBD5VJPED644BZKSCLE/
