as it is, this PEP defers the concept of a "Direct Reference URL" to PEP440.
but then PEP440 partially defers to PEP426's "source_url" concept, when it says "a direct URL reference may be a valid source_url entry" do we expect PEP440 to be updated to fully own what a "Direct Reference URL" can be?, since referring to PEP426 is now a dead-end path (and partially replaced by this PEP) On Mon, Nov 16, 2015 at 12:46 PM, Robert Collins <[email protected]> wrote: > :PEP: XX > :Title: Dependency specification for Python Software Packages > :Version: $Revision$ > :Last-Modified: $Date$ > :Author: Robert Collins <[email protected]> > :BDFL-Delegate: Donald Stufft <[email protected]> > :Discussions-To: distutils-sig <[email protected]> > :Status: Draft > :Type: Standards Track > :Content-Type: text/x-rst > :Created: 11-Nov-2015 > :Post-History: XX > > > Abstract > ======== > > This PEP specifies the language used to describe dependencies for packages. > It draws a border at the edge of describing a single dependency - the > different sorts of dependencies and when they should be installed is a > higher > level problem. The intent is provide a building block for higher layer > specifications. > > The job of a dependency is to enable tools like pip [#pip]_ to find the > right > package to install. Sometimes this is very loose - just specifying a name, > and > sometimes very specific - referring to a specific file to install. > Sometimes > dependencies are only relevant in one platform, or only some versions are > acceptable, so the language permits describing all these cases. > > The language defined is a compact line based format which is already in > widespread use in pip requirements files, though we do not specify the > command > line option handling that those files permit. There is one caveat - the > URL reference form, specified in PEP-440 [#pep440]_ is not actually > implemented in pip, but since PEP-440 is accepted, we use that format > rather > than pip's current native format. > > Motivation > ========== > > Any specification in the Python packaging ecosystem that needs to consume > lists of dependencies needs to build on an approved PEP for such, but > PEP-426 [#pep426]_ is mostly aspirational - and there are already existing > implementations of the dependency specification which we can instead adopt. > The existing implementations are battle proven and user friendly, so > adopting > them is arguably much better than approving an aspirational, unconsumed, > format. > > Specification > ============= > > Examples > -------- > > All features of the language shown with a name based lookup:: > > requests [security,tests] >= 2.8.1, == 2.8.* ; python_version < > "2.7.10" > > A minimal URL based lookup:: > > pip @ > https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686 > > Concepts > -------- > > A dependency specification always specifies a distribution name. It may > include extras, which expand the dependencies of the named distribution to > enable optional features. The version installed can be controlled using > version limits, or giving the URL to a specific artifact to install. > Finally > the dependency can be made conditional using environment markers. > > Grammar > ------- > > We first cover the grammar briefly and then drill into the semantics of > each > section later. > > A distribution specification is written in ASCII text. We use a parsley > [#parsley]_ grammar to provide a precise grammar. It is expected that the > specification will be embedded into a larger system which offers framing > such > as comments, multiple line support via continuations, or other such > features. > > The full grammar including annotations to build a useful parse tree is > included at the end of the PEP. > > Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note: > URI is defined in std-66 [#std66]_:: > > version_cmp = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | > '===' > version = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+ > version_one = version_cmp version wsp* > version_many = version_one (wsp* ',' version_one)* > versionspec = ( '(' version_many ')' ) | version_many > urlspec = '@' wsp* <URI_reference> > > Environment markers allow making a specification only take effect in some > environments:: > > marker_op = version_cmp | 'in' | 'not' wsp+ 'in' > python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' | > '-' | '_' | '*') > dquote = '"' > squote = '\\'' > python_str = (squote (python_str_c | dquote)* squote | > dquote (python_str_c | squote)* dquote) > env_var = ('python_version' | 'python_full_version' | > 'os_name' | 'sys_platform' | 'platform_release' | > 'platform_system' | 'platform_version' | > 'platform_machine' | 'python_implementation' | > 'implementation_name' | 'implementation_version' | > 'extra' # ONLY when defined by a containing layer > ) > marker_var = env_var | python_str > marker_expr = ('(' wsp* marker wsp* ')' > | (marker_var wsp* marker_op wsp* marker_var)) > marker = wsp* marker_expr ( wsp* ('and' | 'or') wsp* > marker_expr)* > quoted_marker = ';' wsp* marker > > Optional components of a distribution may be specified using the extras > field:: > > identifier = letterOrDigit ( > letterOrDigit | > (( letterOrDigit | '-' | '_' | '.')* letterOrDigit ) )* > name = identifier > extras_list = identifier (wsp* ',' wsp* identifier)* > extras = '[' wsp* extras_list? wsp* ']' > > Giving us a rule for name based requirements:: > > name_req = name wsp* extras? wsp* versionspec? wsp* quoted_marker? > > And a rule for direct reference specifications:: > > url_req = name wsp* extras? wsp* urlspec wsp+ quoted_marker? > > Leading to the unified rule that can specify a dependency.:: > > specification = wsp* ( url_req | name_req ) wsp* > > Whitespace > ---------- > > Non line-breaking whitespace is mostly optional with no semantic meaning. > The > sole exception is detecting the end of a URL requirement. > > Names > ----- > > Python distribution names are currently defined in PEP-345 [#pep345]_. > Names > act as the primary identifier for distributions. They are present in all > dependency specifications, and are sufficient to be a specification on > their > own. However, PyPI places strict restrictions on names - they must match a > case insensitive regex or they won't be accepted. Accordingly in this PEP > we > limit the acceptable values for identifiers to that regex. A full > redefinition > of name may take place in a future metadata PEP:: > > ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$ > > Extras > ------ > > An extra is an optional part of a distribution. Distributions can specify > as > many extras as they wish, and each extra results in the declaration of > additional dependencies of the distribution **when** the extra is used in a > dependency specification. For instance:: > > requests[security] > > Extras union in the dependencies they define with the dependencies of the > distribution they are attached to. The example above would result in > requests > being installed, and requests own dependencies, and also any dependencies > that > are listed in the "security" extra of requests. > > If multiple extras are listed, all the dependencies are unioned together. > > Versions > -------- > > See PEP-440 [#pep440]_ for more detail on both version numbers and version > comparisons. Version specifications limit the versions of a distribution > that > can be used. They only apply to distributions looked up by name, rather > than > via a URL. Version comparison are also used in the markers feature. The > optional brackets around a version are present for compatibility with > PEP-345 > [#pep345]_ but should not be generated, only accepted. > > Environment Markers > ------------------- > > Environment markers allow a dependency specification to provide a rule that > describes when the dependency should be used. For instance, consider a > package > that needs argparse. In Python 2.7 argparse is always present. On older > Python > versions it has to be installed as a dependency. This can be expressed as > so:: > > argparse;python_version<"2.7" > > A marker expression evalutes to either True or False. When it evaluates to > False, the dependency specification should be ignored. > > The marker language is a subset of Python itself, chosen for the ability to > safely evaluate it without running arbitrary code that could become a > security > vulnerability. Markers were first standardised in PEP-345 [#pep345]_. This > PEP > fixes some issues that were observed in the design described in PEP-426 > [#pep426]_. > > Comparisons in marker expressions are typed by the comparison operator. > The > <marker_op> operators that are not in <version_cmp> perform the same as > they > do for strings in Python. The <version_cmp> operators use the PEP-440 > [#pep440]_ version comparison rules when those are defined (that is when > both > sides have a valid version specifier). If there is no defined PEP-440 > behaviour and the operator exists in Python, then the operator falls back > to > the Python behaviour. Otherwise an error should be raised. e.g. the > following > will result in errors:: > > "dog" ~= "fred" > python_version ~= "surprise" > > User supplied constants are always encoded as strings with either ``'`` or > ``"`` quote marks. Note that backslash escapes are not defined, but > existing > implementations do support them. They are not included in this > specification because they add complexity and there is no observable need > for > them today. Similarly we do not define non-ASCII character support: all the > runtime variables we are referencing are expected to be ASCII-only. > > The variables in the marker grammar such as "os_name" resolve to values > looked > up in the Python runtime. With the exception of "extra" all values are > defined > on all Python versions today - it is an error in the implementation of > markers > if a value is not defined. > > Unknown variables must raise an error rather than resulting in a comparison > that evaluates to True or False. > > Variables whose value cannot be calculated on a given Python implementation > should evaluate to ``0`` for versions, and an empty string for all other > variables. > > The "extra" variable is special. It is used by wheels to signal which > specifications apply to a given extra in the wheel ``METADATA`` file, but > since the ``METADATA`` file is based on a draft version of PEP-426, there > is > no current specification for this. Regardless, outside of a context where > this > special handling is taking place, the "extra" variable should result in an > error like all other unknown variables. > > .. list-table:: > :header-rows: 1 > > * - Marker > - Python equivalent > - Sample values > * - ``os_name`` > - ``os.name`` > - ``posix``, ``java`` > * - ``sys_platform`` > - ``sys.platform`` > - ``linux``, ``linux2``, ``darwin``, ``java1.8.0_51`` (note that > "linux" > is from Python3 and "linux2" from Python2) > * - ``platform_machine`` > - ``platform.machine()`` > - ``x86_64`` > * - ``python_implementation`` > - ``platform.python_implementation()`` > - ``CPython``, ``Jython`` > * - ``platform_release`` > - ``platform.release()`` > - ``3.14.1-x86_64-linode39``, ``14.5.0``, ``1.8.0_51`` > * - ``platform_system`` > - ``platform.system()`` > - ``Linux``, ``Windows``, ``Java`` > * - ``platform_version`` > - ``platform.version()`` > - ``#1 SMP Fri Apr 25 13:07:35 EDT 2014`` > ``Java HotSpot(TM) 64-Bit Server VM, 25.51-b03, Oracle Corporation`` > ``Darwin Kernel Version 14.5.0: Wed Jul 29 02:18:53 PDT 2015; > root:xnu-2782.40.9~2/RELEASE_X86_64`` > * - ``python_version`` > - ``platform.python_version()[:3]`` > - ``3.4``, ``2.7`` > * - ``python_full_version`` > - ``platform.python_version()`` > - ``3.4.0``, ``3.5.0b1`` > * - ``implementation_name`` > - ``sys.implementation.name`` > - ``cpython`` > * - ``implementation_version`` > - see definition below > - ``3.4.0``, ``3.5.0b1`` > * - ``extra`` > - An error except when defined by the context interpreting the > specification. > - ``test`` > > The ``implementation_version`` marker variable is derived from > ``sys.implementation.version``:: > > def format_full_version(info): > version = '{0.major}.{0.minor}.{0.micro}'.format(info) > kind = info.releaselevel > if kind != 'final': > version += kind[0] + str(info.serial) > return version > > if hasattr(sys, 'implementation'): > implementation_version = > format_full_version(sys.implementation.version) > else: > implementation_version = "0" > > Backwards Compatibility > ======================= > > Most of this PEP is already widely deployed and thus offers no > compatibiltiy > concerns. > > There are however a few points where the PEP differs from the deployed > base. > > Firstly, PEP-440 direct references haven't actually been deployed in the > wild, > but they were designed to be compatibly added, and there are no known > obstacles to adding them to pip or other tools that consume the existing > dependency metadata in distributions - particularly since they won't be > permitted to be present in PyPI uploaded distributions anyway. > > Secondly, PEP-426 markers which have had some reasonable deployment, > particularly in wheels and pip, will handle version comparisons with > ``python_version`` "2.7.10" differently. Specifically in 426 "2.7.10" is > less > than "2.7.9". This backward incompatibility is deliberate. We are also > defining new operators - "~=" and "===", and new variables - > ``platform_release``, ``platform_system``, ``implementation_name``, and > ``implementation_version`` which are not present in older marker > implementations. The variables will error on those implementations. Users > of > both features will need to make a judgement as to when support has become > sufficiently widespread in the ecosystem that using them will not cause > compatibility issues. > > Thirdly, PEP-345 required brackets around version specifiers. In order to > accept PEP-345 dependency specifications, brackets are accepted, but they > should not be generated. > > Rationale > ========= > > In order to move forward with any new PEPs that depend on environment > markers, > we needed a specification that included them in their modern form. This PEP > brings together all the currently unspecified components into a specified > form. > > The requirement specifier was adopted from the EBNF in the setuptools > pkg_resources documentation, since we wish to avoid depending on a > defacto, vs > PEP specified, standard. > > Complete Grammar > ================ > > The complete parsley grammar:: > > wsp = ' ' | '\t' > version_cmp = wsp* <'<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | > '==='> > version = wsp* <( letterOrDigit | '-' | '_' | '.' | '*' | > '+' | '!' )+> > version_one = version_cmp:op version:v wsp* -> (op, v) > version_many = version_one:v1 (wsp* ',' version_one)*:v2 -> [v1] + v2 > versionspec = ('(' version_many:v ')' ->v) | version_many > urlspec = '@' wsp* <URI_reference> > marker_op = version_cmp | 'in' | 'not' wsp+ 'in' > python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' | > '-' | '_' | '*' | '#') > dquote = '"' > squote = '\\'' > python_str = (squote <(python_str_c | dquote)*>:s squote | > dquote <(python_str_c | squote)*>:s dquote) -> s > env_var = ('python_version' | 'python_full_version' | > 'os_name' | 'sys_platform' | 'platform_release' | > 'platform_system' | 'platform_version' | > 'platform_machine' | 'python_implementation' | > 'implementation_name' | 'implementation_version' | > 'extra' # ONLY when defined by a containing layer > ):varname -> lookup(varname) > marker_var = env_var | python_str > marker_expr = (("(" wsp* marker:m wsp* ")" -> m) > | ((marker_var:l wsp* marker_op:o wsp* > marker_var:r)) > -> (l, o, r)) > marker = (wsp* marker_expr:m ( wsp* ("and" | "or"):o wsp* > marker_expr:r -> (o, r))*:ms -> (m, ms)) > quoted_marker = ';' wsp* marker > identifier = <letterOrDigit ( > letterOrDigit | > (( letterOrDigit | '-' | '_' | '.')* letterOrDigit ) > )*> > name = identifier > extras_list = identifier:i (wsp* ',' wsp* identifier)*:ids -> [i] + > ids > extras = '[' wsp* extras_list?:e wsp* ']' -> e > name_req = (name:n wsp* extras?:e wsp* versionspec?:v wsp* > quoted_marker?:m > -> (n, e or [], v or [], m)) > url_req = (name:n wsp* extras?:e wsp* urlspec:v wsp+ > quoted_marker?:m > -> (n, e or [], v, m)) > specification = wsp* ( url_req | name_req ):s wsp* -> s > # The result is a tuple - name, list-of-extras, > # list-of-version-constraints-or-a-url, marker-ast or None > > > URI_reference = <URI | relative_ref> > URI = scheme ':' hier_part ('?' query )? ( '#' fragment)? > hier_part = ('//' authority path_abempty) | path_absolute | > path_rootless | path_empty > absolute_URI = scheme ':' hier_part ( '?' query )? > relative_ref = relative_part ( '?' query )? ( '#' fragment )? > relative_part = '//' authority path_abempty | path_absolute | > path_noscheme | path_empty > scheme = letter ( letter | digit | '+' | '-' | '.')* > authority = ( userinfo '@' )? host ( ':' port )? > userinfo = ( unreserved | pct_encoded | sub_delims | ':')* > host = IP_literal | IPv4address | reg_name > port = digit* > IP_literal = '[' ( IPv6address | IPvFuture) ']' > IPvFuture = 'v' hexdig+ '.' ( unreserved | sub_delims | ':')+ > IPv6address = ( > ( h16 ':'){6} ls32 > | '::' ( h16 ':'){5} ls32 > | ( h16 )? '::' ( h16 ':'){4} ls32 > | ( ( h16 ':')? h16 )? '::' ( h16 ':'){3} ls32 > | ( ( h16 ':'){0,2} h16 )? '::' ( h16 ':'){2} ls32 > | ( ( h16 ':'){0,3} h16 )? '::' h16 ':' ls32 > | ( ( h16 ':'){0,4} h16 )? '::' ls32 > | ( ( h16 ':'){0,5} h16 )? '::' h16 > | ( ( h16 ':'){0,6} h16 )? '::' ) > h16 = hexdig{1,4} > ls32 = ( h16 ':' h16) | IPv4address > IPv4address = dec_octet '.' dec_octet '.' dec_octet '.' Dec_octet > nz = ~'0' digit > dec_octet = ( > digit # 0-9 > | nz digit # 10-99 > | '1' digit{2} # 100-199 > | '2' ('0' | '1' | '2' | '3' | '4') digit # 200-249 > | '25' ('0' | '1' | '2' | '3' | '4' | '5') )# > %250-255 > reg_name = ( unreserved | pct_encoded | sub_delims)* > path = ( > path_abempty # begins with '/' or is empty > | path_absolute # begins with '/' but not '//' > | path_noscheme # begins with a non-colon segment > | path_rootless # begins with a segment > | path_empty ) # zero characters > path_abempty = ( '/' segment)* > path_absolute = '/' ( segment_nz ( '/' segment)* )? > path_noscheme = segment_nz_nc ( '/' segment)* > path_rootless = segment_nz ( '/' segment)* > path_empty = pchar{0} > segment = pchar* > segment_nz = pchar+ > segment_nz_nc = ( unreserved | pct_encoded | sub_delims | '@')+ > # non-zero-length segment without any colon ':' > pchar = unreserved | pct_encoded | sub_delims | ':' | '@' > query = ( pchar | '/' | '?')* > fragment = ( pchar | '/' | '?')* > pct_encoded = '%' hexdig > unreserved = letter | digit | '-' | '.' | '_' | '~' > reserved = gen_delims | sub_delims > gen_delims = ':' | '/' | '?' | '#' | '(' | ')?' | '@' > sub_delims = '!' | '$' | '&' | '\\'' | '(' | ')' | '*' | '+' | > ',' | ';' | '=' > hexdig = digit | 'a' | 'A' | 'b' | 'B' | 'c' | 'C' | 'd' | > 'D' | 'e' | 'E' | 'f' | 'F' > > A test program - if the grammar is in a string ``grammar``:: > > import os > import sys > import platform > > from parsley import makeGrammar > > grammar = """ > wsp ... > """ > tests = [ > "A", > "aa", > "name", > "name>=3", > "name>=3,<2", > "name [fred,bar] @ http://foo.com ; python_version=='2.7'", > "name[quux, strange];python_version<'2.7' and > platform_version=='2'", > "name; os_name=='dud' and (os_name=='odd' or os_name=='fred')", > "name; os_name=='dud' and os_name=='odd' or os_name=='fred'", > ] > > def format_full_version(info): > version = '{0.major}.{0.minor}.{0.micro}'.format(info) > kind = info.releaselevel > if kind != 'final': > version += kind[0] + str(info.serial) > return version > > if hasattr(sys, 'implementation'): > implementation_version = > format_full_version(sys.implementation.version) > implementation_name = sys.implementation.name > else: > implementation_version = '0' > implementation_name = '' > bindings = { > 'implementation_name': implementation_name, > 'implementation_version': implementation_version, > 'os_name': os.name, > 'platform_machine': platform.machine(), > 'platform_release': platform.release(), > 'platform_system': platform.system(), > 'platform_version': platform.version(), > 'python_full_version': platform.python_version(), > 'python_implementation': platform.python_implementation(), > 'python_version': platform.python_version()[:3], > 'sys_platform': sys.platform, > } > > compiled = makeGrammar(grammar, {'lookup': bindings.__getitem__}) > for test in tests: > parsed = compiled(test).specification() > print(parsed) > > References > ========== > > .. [#pip] pip, the recommended installer for Python packages > (http://pip.readthedocs.org/en/stable/) > > .. [#pep345] PEP-345, Python distribution metadata version 1.2. > (https://www.python.org/dev/peps/pep-0345/) > > .. [#pep426] PEP-426, Python distribution metadata. > (https://www.python.org/dev/peps/pep-0426/) > > .. [#pep440] PEP-440, Python distribution metadata. > (https://www.python.org/dev/peps/pep-0440/) > > .. [#std66] The URL specification. > (https://tools.ietf.org/html/rfc3986) > > .. [#parsley] The parsley PEG library. > (https://pypi.python.org/pypi/parsley/) > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > -- > Robert Collins <[email protected]> > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - [email protected] > https://mail.python.org/mailman/listinfo/distutils-sig >
_______________________________________________ Distutils-SIG maillist - [email protected] https://mail.python.org/mailman/listinfo/distutils-sig
