Re: Adding TEST_ENV/UTF-8 to various ports
On 2019/03/17 20:24, Kurt Mosiejczuk wrote:
> On Sun, Mar 17, 2019 at 11:12:29PM +, Stuart Henderson wrote:
>
> > Instead of making "do-test" conditional, just make MODPY_TEST_CMD
> > conditional instead. It's then carried into MODPY_TEST_TARGET which can
> > stay the same in both cases. No need for a special MODPY_PYTEST_ARGS
> > either; the existing TEST_TARGET does the job nicely. And extend the
> > conditional around do-test, rather than adding a second branch.
> > Untested but I think something like this should do the trick:
>
> > .if ${MODPY_PYTEST:L} == "yes"
> > MODPY_TEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
> > ${MODPY_BIN} -m pytest
> > .else
> > MODPY_TEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
> > ${MODPY_BIN} ./${MODPY_SETUP} \
> > ${MODPY_SETUP_ARGS}
> > .endif
>
> > ...
>
> > . if !target(do-test) && (${MODPY_SETUPUTILS:L} == "yes" ||
> > ${MODPY_PYTEST:L} == "yes")
> > do-test:
> > @${MODPY_TEST_TARGET}
> > . endif
>
> The problem I'm having right now is TEST_TARGETS gets defined as test if it
> is empty. pytest does not use this as a default. I'm testing with an
> adapted devel/py-dateutil and tests just fail with:
>
> = no tests ran in 0.01 seconds
> =
> ERROR: file not found: test
>
> I've included my current diff for our final third. It is a diff after the
> first two parts have been applied. I'm also including a diff for py-dateutil.
>
> In the default case with pytest we *don't* want a TEST_TARGET to happen but
> if we leave it empty it gets filled in. We don't want to override it to
> empty it if it has been set. So we'd be back at some sort of dance with
> MODPY_PYTEST_ARGS. I may be missing something though.
Ah OK. In that case, MODPY_PYTEST_ARGS (empty by default, with ?=) makes sense.
> Index: Makefile
> ===
> RCS file: /cvs/ports/devel/py-dateutil/Makefile,v
> retrieving revision 1.28
> diff -u -p -r1.28 Makefile
> --- Makefile 3 Mar 2019 08:28:31 - 1.28
> +++ Makefile 18 Mar 2019 00:17:11 -
> @@ -18,21 +18,15 @@ MODULES = lang/python
>
> MODPY_PI = Yes
> MODPY_SETUPTOOLS = Yes
> +MODPY_PYTEST = Yes
>
> BUILD_DEPENDS = devel/py-setuptools_scm${MODPY_FLAVOR}
> RUN_DEPENDS =devel/py-six${MODPY_FLAVOR}
> TEST_DEPENDS = devel/py-freezegun${MODPY_FLAVOR} \
> devel/py-hypothesis${MODPY_FLAVOR} \
> - devel/py-six${MODPY_FLAVOR} \
> - devel/py-test${MODPY_FLAVOR}
> + devel/py-six${MODPY_FLAVOR}
>
> FLAVORS =python3
> FLAVOR ?=
> -
> -TEST_ENV += LC_CTYPE=C.UTF-8
> -
> -do-test:
> - cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
> - ${MODPY_BIN} -m pytest
>
> .include
> --- python.port.mk.phase2 Sun Mar 17 19:59:34 2019
> +++ python.port.mkSun Mar 17 19:59:56 2019
> @@ -63,12 +63,18 @@ MODPY_PYOEXTENSION = pyo
> MODPY_PYOEXTENSION ?=opt-1.pyc
> .endif
>
> -MODPY_WANTLIB = python${MODPY_VERSION}${MODPY_LIB_SUFFIX}
> +MODPY_WANTLIB = python${MODPY_VERSION}${MODPY_LIB_SUFFIX}
>
> +MODPY_PYTEST ?= no
> +
> MODPY_RUN_DEPENDS = lang/python/${MODPY_VERSION}
> MODPY_LIB_DEPENDS = lang/python/${MODPY_VERSION}
> _MODPY_BUILD_DEPENDS = lang/python/${MODPY_VERSION}
> +.if ${MODPY_PYTEST:L} == "yes"
> +MODPY_TEST_DEPENDS = devel/py-test${MODPY_FLAVOR}
> +.else
> MODPY_TEST_DEPENDS =
> +.endif
>
> .if ${NO_BUILD:L} == "no"
> MODPY_BUILDDEP ?=Yes
> @@ -161,9 +167,14 @@ MODPY_CMD = cd ${WRKSRC} && ${SETENV} ${MAKE_ENV} \
> ${MODPY_BIN} ./${MODPY_SETUP} \
> ${MODPY_SETUP_ARGS}
>
> +.if ${MODPY_PYTEST:L} == "yes"
> MODPY_TEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
> + ${MODPY_BIN} -m pytest
> +.else
> +MODPY_TEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
> ${MODPY_BIN} ./${MODPY_SETUP} \
> ${MODPY_SETUP_ARGS}
> +.endif
>
> MODPY_TEST_LOCALE ?= LC_CTYPE=en_US.UTF-8
>
> @@ -209,7 +220,8 @@ do-install:
> . endif
>
> # setuptools supports regress testing from setup.py using a standard target
> -. if !target(do-test) && ${MODPY_SETUPUTILS:L} == "yes"
> +. if !target(do-test) && (${MODPY_SETUPUTILS:L} == "yes" || \
> + ${MODPY_PYTEST:L} == "yes" )
> do-test:
> @${MODPY_TEST_TARGET}
> . endif
Re: Adding TEST_ENV/UTF-8 to various ports
On 2019/03/17 16:22, Kurt Mosiejczuk wrote:
> On Sun, Mar 17, 2019 at 02:10:47PM -0400, Kurt Mosiejczuk wrote:
>
> > Here are diffs for python.port.mk and port-modules.5.
>
> > I changed Remi's diff a bit. With the discussion about C.UTF-8 being
> > an alias for en_US.UTF-8, I changed the default locale we add to that.
> > Additionally, I changed it so it is MODPY_TEST_LOCALE (that defaults to
> > en_US.UTF-8) that gets added to TEST_ENV.
>
> > Similar to MODPY_SETUPTOOLS, I have MODPY_PYTEST adding devel/py-test
> > to TEST_DEPENDS. I added code to do that and added a MODPY_TEST_DEPENDS
> > that is similar to MODPY_*_DEPENDS that will keep us from adding
> > anything to TEST_DEPENDS if it is set to No.
>
> > I added all these new environment variables to port-modules.5, which is
> > the second diff included here.
>
> Here is a fixed diff for python.port.mk
>
> I need to not make tweaks after tests.
I like the direction this is taking, but it's missing use of a standard
MODULES mechanism, and a comment on style for commit -
> --Kurt
>
> Index: python.port.mk
> ===
> RCS file: /cvs/ports/lang/python/python.port.mk,v
> retrieving revision 1.100
> diff -u -p -r1.100 python.port.mk
> --- python.port.mk4 Dec 2018 05:57:31 - 1.100
> +++ python.port.mk17 Mar 2019 20:21:45 -
> @@ -68,6 +68,9 @@ MODPY_WANTLIB = python${MODPY_VERSION}${
> MODPY_RUN_DEPENDS = lang/python/${MODPY_VERSION}
> MODPY_LIB_DEPENDS = lang/python/${MODPY_VERSION}
> _MODPY_BUILD_DEPENDS = lang/python/${MODPY_VERSION}
> +.if defined(MODPY_TEST) && ${MODPY_PYTEST:L} == "yes"
> +MODPY_TEST_DEPENDS = devel/py-test
> +.endif
>
> .if ${NO_BUILD:L} == "no"
> MODPY_BUILDDEP ?=Yes
> @@ -75,6 +78,7 @@ MODPY_BUILDDEP ?= Yes
> MODPY_BUILDDEP ?=No
> .endif
> MODPY_RUNDEP ?= Yes
> +MODPY_TESTDEP ?= Yes
>
> .if ${MODPY_BUILDDEP:L} == "yes"
> BUILD_DEPENDS += ${_MODPY_BUILD_DEPENDS}
> @@ -84,6 +88,10 @@ BUILD_DEPENDS += ${_MODPY_BUILD_DEPENDS}
> RUN_DEPENDS += ${MODPY_RUN_DEPENDS}
> .endif
>
> +.if ${MODPY_TESTDEP:L} == "yes'
> +TEST_DEPENDS += ${MODPY_TEST_DEPENDS}
> +.endif
There are several logically separate pieces here which would be
better to commit independently to keep the history clear.
Adding MODPY_TESTDEP is one part which definitely stands by itself.
I think MODPY_TEST_LOCALE also stands alone.
And the pytest parts are a third piece.
> +
> _MODPY_PRE_BUILD_STEPS = :
> .if defined(MODPY_SETUPTOOLS) && ${MODPY_SETUPTOOLS:L} == "yes"
> # The setuptools module provides a package locator (site.py) that is
> @@ -159,6 +167,13 @@ MODPY_TEST_CMD = cd ${WRKSRC} && ${SETEN
> ${MODPY_BIN} ./${MODPY_SETUP} \
> ${MODPY_SETUP_ARGS}
>
> +MODPY_PYTEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
> + ${MODPY_BIN} -m pytest
> +
> +MODPY_TEST_LOCALE ?= LC_CTYPE=en_US.UTF-8
> +
> +TEST_ENV += ${MODPY_TEST_LOCALE}
> +
> SUBST_VARS :=MODPY_PYCACHE MODPY_COMMENT MODPY_ABI3SO
> MODPY_PYC_MAGIC_TAG \
> MODPY_BIN MODPY_EGG_VERSION MODPY_VERSION MODPY_BIN_SUFFIX \
> MODPY_PY_PREFIX MODPY_PYOEXTENSION ${SUBST_VARS}
> @@ -184,6 +199,8 @@ MODPY_INSTALL_TARGET = \
> ${MODPY_DISTUTILS_INSTALL} ${MODPY_DISTUTILS_INSTALLARGS}
> MODPY_TEST_TARGET = \
> ${MODPY_TEST_CMD} ${TEST_TARGET}
> +MODPY_PYTEST ?= no
> +MODPY_PYTEST_ARGS ?=
>
> # dirty way to do it with no modifications in bsd.port.mk
> .if empty(CONFIGURE_STYLE)
> @@ -198,8 +215,11 @@ do-install:
> @${MODPY_INSTALL_TARGET}
> . endif
>
> +. if !target(do-test) && ${MODPY_PYTEST:L} == "yes"
> +do-test:
> + ${MODPY_PYTEST_CMD} ${MODPY_PYTEST_ARGS}
> # setuptools supports regress testing from setup.py using a standard target
> -. if !target(do-test) && ${MODPY_SETUPUTILS:L} == "yes"
> +. elif !target(do-test) && ${MODPY_SETUPUTILS:L} == "yes"
> do-test:
> @${MODPY_TEST_TARGET}
> . endif
MODxxx_TEST_TARGET is a standard mechanism used in various modules.
Sometimes a port has a special do-test target but still wants to use the
standard one as part of it, or uses multiple MODULES and wants to pick
a test target from just one of them. So it would be better to continue
making this mechanism available in the pytest case too.
Instead of making "do-test" conditional, just make MODPY_TEST_CMD
conditional instead. It's then carried into MODPY_TEST_TARGET which can
stay the same in both cases. No need for a special MODPY_PYTEST_ARGS
either; the existing TEST_TARGET does the job nicely. And extend the
conditional around do-test, rather than adding a second branch.
Untested but I think something like this should do the trick:
.if ${MODPY_PYTEST:L} == "yes"
MODPY_TEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
${MODPY_BIN} -m pytest
.else
Re: Adding TEST_ENV/UTF-8 to various ports
On Sun, Mar 17, 2019 at 04:22:51PM -0400, Kurt Mosiejczuk wrote:
> On Sun, Mar 17, 2019 at 02:10:47PM -0400, Kurt Mosiejczuk wrote:
>
> > Here are diffs for python.port.mk and port-modules.5.
>
> > I changed Remi's diff a bit. With the discussion about C.UTF-8 being
> > an alias for en_US.UTF-8, I changed the default locale we add to that.
> > Additionally, I changed it so it is MODPY_TEST_LOCALE (that defaults to
> > en_US.UTF-8) that gets added to TEST_ENV.
>
> > Similar to MODPY_SETUPTOOLS, I have MODPY_PYTEST adding devel/py-test
> > to TEST_DEPENDS. I added code to do that and added a MODPY_TEST_DEPENDS
> > that is similar to MODPY_*_DEPENDS that will keep us from adding
> > anything to TEST_DEPENDS if it is set to No.
>
I think adding devel/py-test to TEST_DEPENDS is useful.
Some comments below.
> > I added all these new environment variables to port-modules.5, which is
> > the second diff included here.
>
> Here is a fixed diff for python.port.mk
>
> I need to not make tweaks after tests.
>
> --Kurt
>
> Index: python.port.mk
> ===
> RCS file: /cvs/ports/lang/python/python.port.mk,v
> retrieving revision 1.100
> diff -u -p -r1.100 python.port.mk
> --- python.port.mk4 Dec 2018 05:57:31 - 1.100
> +++ python.port.mk17 Mar 2019 20:21:45 -
> @@ -68,6 +68,9 @@ MODPY_WANTLIB = python${MODPY_VERSION}${
> MODPY_RUN_DEPENDS = lang/python/${MODPY_VERSION}
> MODPY_LIB_DEPENDS = lang/python/${MODPY_VERSION}
> _MODPY_BUILD_DEPENDS = lang/python/${MODPY_VERSION}
> +.if defined(MODPY_TEST) && ${MODPY_PYTEST:L} == "yes"
Should this be "defined(MODPY_*PY*TEST)"? Why not move "MODPY_PYTEST ?= no"
up and get rid of the "defined" test?
> +MODPY_TEST_DEPENDS = devel/py-test
Does this also work for python3 ports? I think you need to append
${MODPY_FLAVOR}.
> +.endif
>
> .if ${NO_BUILD:L} == "no"
> MODPY_BUILDDEP ?=Yes
> @@ -75,6 +78,7 @@ MODPY_BUILDDEP ?= Yes
> MODPY_BUILDDEP ?=No
> .endif
> MODPY_RUNDEP ?= Yes
> +MODPY_TESTDEP ?= Yes
>
> .if ${MODPY_BUILDDEP:L} == "yes"
> BUILD_DEPENDS += ${_MODPY_BUILD_DEPENDS}
> @@ -84,6 +88,10 @@ BUILD_DEPENDS += ${_MODPY_BUILD_DEPENDS}
> RUN_DEPENDS += ${MODPY_RUN_DEPENDS}
> .endif
>
> +.if ${MODPY_TESTDEP:L} == "yes'
> +TEST_DEPENDS += ${MODPY_TEST_DEPENDS}
> +.endif
> +
> _MODPY_PRE_BUILD_STEPS = :
> .if defined(MODPY_SETUPTOOLS) && ${MODPY_SETUPTOOLS:L} == "yes"
> # The setuptools module provides a package locator (site.py) that is
> @@ -159,6 +167,13 @@ MODPY_TEST_CMD = cd ${WRKSRC} && ${SETEN
> ${MODPY_BIN} ./${MODPY_SETUP} \
> ${MODPY_SETUP_ARGS}
>
> +MODPY_PYTEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
> + ${MODPY_BIN} -m pytest
> +
> +MODPY_TEST_LOCALE ?= LC_CTYPE=en_US.UTF-8
> +
> +TEST_ENV += ${MODPY_TEST_LOCALE}
> +
> SUBST_VARS :=MODPY_PYCACHE MODPY_COMMENT MODPY_ABI3SO
> MODPY_PYC_MAGIC_TAG \
> MODPY_BIN MODPY_EGG_VERSION MODPY_VERSION MODPY_BIN_SUFFIX \
> MODPY_PY_PREFIX MODPY_PYOEXTENSION ${SUBST_VARS}
> @@ -184,6 +199,8 @@ MODPY_INSTALL_TARGET = \
> ${MODPY_DISTUTILS_INSTALL} ${MODPY_DISTUTILS_INSTALLARGS}
> MODPY_TEST_TARGET = \
> ${MODPY_TEST_CMD} ${TEST_TARGET}
> +MODPY_PYTEST ?= no
> +MODPY_PYTEST_ARGS ?=
>
> # dirty way to do it with no modifications in bsd.port.mk
> .if empty(CONFIGURE_STYLE)
> @@ -198,8 +215,11 @@ do-install:
> @${MODPY_INSTALL_TARGET}
> . endif
>
> +. if !target(do-test) && ${MODPY_PYTEST:L} == "yes"
> +do-test:
> + ${MODPY_PYTEST_CMD} ${MODPY_PYTEST_ARGS}
> # setuptools supports regress testing from setup.py using a standard target
> -. if !target(do-test) && ${MODPY_SETUPUTILS:L} == "yes"
> +. elif !target(do-test) && ${MODPY_SETUPUTILS:L} == "yes"
> do-test:
> @${MODPY_TEST_TARGET}
> . endif
>
Re: Adding TEST_ENV/UTF-8 to various ports
Hi,
Remi Locherer wrote on Sat, Mar 16, 2019 at 11:26:54PM +0100:
> On Sat, Mar 16, 2019 at 09:48:21PM +, Stuart Henderson wrote:
>> On 2019/03/16 22:12, Remi Locherer wrote:
>>> Index: lang/python/python.port.mk
>>> ===
>>> RCS file: /cvs/ports/lang/python/python.port.mk,v
>>> retrieving revision 1.100
>>> diff -u -p -r1.100 python.port.mk
>>> --- lang/python/python.port.mk 4 Dec 2018 05:57:31 - 1.100
>>> +++ lang/python/python.port.mk 16 Mar 2019 20:40:34 -
>>> @@ -150,6 +150,7 @@ CONFIGURE_ENV +=PYTHON="${MODPY_BIN}"
>>> CONFIGURE_ENV += ac_cv_prog_PYTHON="${MODPY_BIN}" \
>>> ac_cv_path_PYTHON="${MODPY_BIN}"
>>> .endif
>>> +TEST_ENV +=LC_CTYPE=C.UTF-8
>> Do we actually support LC_CTYPE=C.UTF-8?
We do not recommend it; the locale(1) manual page only recommends
either leaving LC_* unset or setting LC_CTYPE=en_US.UTF-8.
But we do support it, just like we support LC_CTYPE=FooBar.UTF-8.
The locale(1) manual page says:
If the value of LC_CTYPE ends in ".UTF-8", programs in the OpenBSD
base system ignore the beginning of it, treating for example
zh_CN.UTF-8 exactly like en_US.UTF-8. Programs from packages(7)
may however make a difference.
Theoretically, it could happen that at some point in the future,
we might stop supporting LC_CTYPE=FooBar.UTF-8, though i do not
expect that. But i don't think we could ever stop supporting
LC_CTYPE=C.UTF-8. It seems too widespread in practice, and removing
support for it would no doubt break more than one thing in ports.
> According to "locale -a" we do.
The -a option of locale(1) is a scam. It should never be used as
an argument for or against anything. Its only purpose is to appease
ports who insist in inspecting it, but even for that purpose, it
is far from perfect.
> I proposed C.UTF-8 because I think this is what python prefers
> after skimming over https://www.python.org/dev/peps/pep-0538/ .
If that's what Python folks like, i think there is nothing wrong with
using it; on OpenBSD, it's just an alias for LC_CTYPE=en_US.UTF-8.
On other operating systems, it may or may not work, exactly like
LC_CTYPE=en_US.UTF-8 may or may not work elsewehere. I have seen
systems where LC_CTYPE=en_US.UTF-8 works and LC_CTYPE=C.UTF-8 does
not and vice versa; i think i have even seen systems where neither
work. Locale names are simply not standardized - except for "C"
and "POSIX".
I didn't speak up earlier because the last time i did a substantial
programming project in Python was about a decade ago, so i'm no
longer qualified to OK or to object to Python patches...
Yours,
Ingo
Re: Adding TEST_ENV/UTF-8 to various ports
On Sat, Mar 16, 2019 at 11:26:54PM +0100, Remi Locherer wrote: > > Do we actually support LC_CTYPE=C.UTF-8? > > > > According to "locale -a" we do. > > I proposed C.UTF-8 because I think this is what python prefers after skimming > over https://www.python.org/dev/peps/pep-0538/ . This; the following ports already use it: devel/py-click-log devel/py-dateutil devel/py-ipython_genutils devel/py-lazy-object-proxy devel/py-path.py devel/py-py net/toot sysutils/py-scandir telephony/py-phonenumbers textproc/py-prettytable textproc/py-unidecode www/jupyter-notebook www/py-flask www/weboob I'm OK with your bsd.port.mk diff, but please update port-modules(5) accordingly.
Re: Adding TEST_ENV/UTF-8 to various ports
On 3/16/19 5:12 PM, Remi Locherer wrote: This adds TEST_ENV to python.port.mk. Plus it adds MODPY_PYTEST and MODPY_PYTEST_ARGS to deal with "-m pytest". I counted a bit more than 70 ports using this testing method. I added productivity/vdirsyncer as an example. OK? Remi Tried on an assortment of relevant ports, including py-click-log, py-cairocffi (needs MODPY_PYTEST_ARGS = lib/cairocffi) and the py3 version of py-lazy-object-proxy, as well as py-betamax and the included vdirsyncer, with no surprises. for my part, OK pamela@
Re: Adding TEST_ENV/UTF-8 to various ports
On Sat, Mar 16, 2019 at 09:48:21PM +, Stuart Henderson wrote:
> On 2019/03/16 22:12, Remi Locherer wrote:
> > This adds TEST_ENV to python.port.mk. Plus it adds MODPY_PYTEST and
> > MODPY_PYTEST_ARGS to deal with "-m pytest". I counted a bit more than
> > 70 ports using this testing method.
> >
> > I added productivity/vdirsyncer as an example.
> >
> > OK?
>
> This looks a sane approach,
>
> > Remi
> >
> >
> > Index: lang/python/python.port.mk
> > ===
> > RCS file: /cvs/ports/lang/python/python.port.mk,v
> > retrieving revision 1.100
> > diff -u -p -r1.100 python.port.mk
> > --- lang/python/python.port.mk 4 Dec 2018 05:57:31 - 1.100
> > +++ lang/python/python.port.mk 16 Mar 2019 20:40:34 -
> > @@ -150,6 +150,7 @@ CONFIGURE_ENV +=PYTHON="${MODPY_BIN}"
> > CONFIGURE_ENV += ac_cv_prog_PYTHON="${MODPY_BIN}" \
> > ac_cv_path_PYTHON="${MODPY_BIN}"
> > .endif
> > +TEST_ENV +=LC_CTYPE=C.UTF-8
>
> Do we actually support LC_CTYPE=C.UTF-8?
>
According to "locale -a" we do.
I proposed C.UTF-8 because I think this is what python prefers after skimming
over https://www.python.org/dev/peps/pep-0538/ .
Re: Adding TEST_ENV/UTF-8 to various ports
On 2019/03/16 22:12, Remi Locherer wrote:
> This adds TEST_ENV to python.port.mk. Plus it adds MODPY_PYTEST and
> MODPY_PYTEST_ARGS to deal with "-m pytest". I counted a bit more than
> 70 ports using this testing method.
>
> I added productivity/vdirsyncer as an example.
>
> OK?
This looks a sane approach,
> Remi
>
>
> Index: lang/python/python.port.mk
> ===
> RCS file: /cvs/ports/lang/python/python.port.mk,v
> retrieving revision 1.100
> diff -u -p -r1.100 python.port.mk
> --- lang/python/python.port.mk4 Dec 2018 05:57:31 - 1.100
> +++ lang/python/python.port.mk16 Mar 2019 20:40:34 -
> @@ -150,6 +150,7 @@ CONFIGURE_ENV += PYTHON="${MODPY_BIN}"
> CONFIGURE_ENV += ac_cv_prog_PYTHON="${MODPY_BIN}" \
> ac_cv_path_PYTHON="${MODPY_BIN}"
> .endif
> +TEST_ENV += LC_CTYPE=C.UTF-8
Do we actually support LC_CTYPE=C.UTF-8?
Re: Adding TEST_ENV/UTF-8 to various ports
On Fri, Mar 15, 2019 at 11:46:30AM +0100, Antoine Jacoutot wrote:
> On Fri, Mar 15, 2019 at 10:43:39AM +, Stuart Henderson wrote:
> > On 2019/03/15 11:25, Antoine Jacoutot wrote:
> > > On Fri, Mar 15, 2019 at 09:48:39AM +, Stuart Henderson wrote:
> > > > I'd really like if this could be factored into the module so we can get
> > > > rid
> > > > of copies all over the tree.. perhaps behind a MODPY_PYTEST=Yes and
> > > > using
> > > > something like MODPY_PYTEST_FLAGS for those that need more than just -m
> > > > pytest?
> > >
> > > Why don't we add it by default for lang/python module ports?
> > > I mean, would it hurt anything?
> >
> > We already have a different default test target for python that uses
> > setup.py.. It's annoying, there are about 3 common methods.
>
> I mean adding a default TEST_ENV in the module.
This adds TEST_ENV to python.port.mk. Plus it adds MODPY_PYTEST and
MODPY_PYTEST_ARGS to deal with "-m pytest". I counted a bit more than
70 ports using this testing method.
I added productivity/vdirsyncer as an example.
OK?
Remi
Index: lang/python/python.port.mk
===
RCS file: /cvs/ports/lang/python/python.port.mk,v
retrieving revision 1.100
diff -u -p -r1.100 python.port.mk
--- lang/python/python.port.mk 4 Dec 2018 05:57:31 - 1.100
+++ lang/python/python.port.mk 16 Mar 2019 20:40:34 -
@@ -150,6 +150,7 @@ CONFIGURE_ENV +=PYTHON="${MODPY_BIN}"
CONFIGURE_ENV += ac_cv_prog_PYTHON="${MODPY_BIN}" \
ac_cv_path_PYTHON="${MODPY_BIN}"
.endif
+TEST_ENV +=LC_CTYPE=C.UTF-8
MODPY_CMD =cd ${WRKSRC} && ${SETENV} ${MAKE_ENV} \
${MODPY_BIN} ./${MODPY_SETUP} \
@@ -159,6 +160,9 @@ MODPY_TEST_CMD = cd ${WRKSRC} && ${SETEN
${MODPY_BIN} ./${MODPY_SETUP} \
${MODPY_SETUP_ARGS}
+MODPY_PYTEST_CMD = cd ${WRKSRC} && ${SETENV} ${ALL_TEST_ENV} \
+ ${MODPY_BIN} -m pytest
+
SUBST_VARS := MODPY_PYCACHE MODPY_COMMENT MODPY_ABI3SO MODPY_PYC_MAGIC_TAG \
MODPY_BIN MODPY_EGG_VERSION MODPY_VERSION MODPY_BIN_SUFFIX \
MODPY_PY_PREFIX MODPY_PYOEXTENSION ${SUBST_VARS}
@@ -184,6 +188,8 @@ MODPY_INSTALL_TARGET = \
${MODPY_DISTUTILS_INSTALL} ${MODPY_DISTUTILS_INSTALLARGS}
MODPY_TEST_TARGET = \
${MODPY_TEST_CMD} ${TEST_TARGET}
+MODPY_PYTEST ?=no
+MODPY_PYTEST_ARGS ?=
# dirty way to do it with no modifications in bsd.port.mk
.if empty(CONFIGURE_STYLE)
@@ -198,8 +204,11 @@ do-install:
@${MODPY_INSTALL_TARGET}
. endif
+. if !target(do-test) && ${MODPY_PYTEST:L} == "yes"
+do-test:
+ ${MODPY_PYTEST_CMD} ${MODPY_PYTEST_ARGS}
# setuptools supports regress testing from setup.py using a standard target
-. if !target(do-test) && ${MODPY_SETUPUTILS:L} == "yes"
+. elif !target(do-test) && ${MODPY_SETUPUTILS:L} == "yes"
do-test:
@${MODPY_TEST_TARGET}
. endif
Index: productivity/vdirsyncer/Makefile
===
RCS file: /cvs/ports/productivity/vdirsyncer/Makefile,v
retrieving revision 1.7
diff -u -p -r1.7 Makefile
--- productivity/vdirsyncer/Makefile16 Aug 2018 21:37:55 - 1.7
+++ productivity/vdirsyncer/Makefile16 Mar 2019 20:40:34 -
@@ -20,6 +20,8 @@ MODULES = lang/python
MODPY_VERSION= ${MODPY_DEFAULT_VERSION_3}
MODPY_PI = Yes
MODPY_SETUPTOOLS = Yes
+MODPY_PYTEST = Yes
+MODPY_PYTEST_ARGS =tests/
BUILD_DEPENDS =textproc/py-sphinx${MODPY_FLAVOR} \
devel/py-setuptools_scm${MODPY_FLAVOR} \
@@ -37,6 +39,9 @@ TEST_DEPENDS =devel/py-hypothesis${MOD
www/py-urllib3${MODPY_FLAVOR} \
${RUN_DEPENDS}
+TEST_ENV +=DAV_SERVER=skip \
+ DETERMINISTIC_TESTS=true
+
post-build:
cd ${WRKSRC}/docs; ${MAKE_ENV} PYTHONPATH=${WRKSRC} \
sphinx-build${MODPY_BIN_SUFFIX} \
@@ -52,9 +57,5 @@ post-install:
pax -rw * ${PREFIX}/share/doc/vdirsyncer
${INSTALL_DATA} ${WRKSRC}/config.example \
${PREFIX}/share/examples/vdirsyncer
-
-do-test:
- cd ${WRKSRC}; DAV_SERVER=skip DETERMINISTIC_TESTS=true \
- ${MODPY_BIN} -m pytest tests/
.include
Re: Adding TEST_ENV/UTF-8 to various ports
On Fri, Mar 15, 2019 at 10:43:39AM +, Stuart Henderson wrote: > On 2019/03/15 11:25, Antoine Jacoutot wrote: > > On Fri, Mar 15, 2019 at 09:48:39AM +, Stuart Henderson wrote: > > > I'd really like if this could be factored into the module so we can get > > > rid > > > of copies all over the tree.. perhaps behind a MODPY_PYTEST=Yes and using > > > something like MODPY_PYTEST_FLAGS for those that need more than just -m > > > pytest? > > > > Why don't we add it by default for lang/python module ports? > > I mean, would it hurt anything? > > We already have a different default test target for python that uses > setup.py.. It's annoying, there are about 3 common methods. I mean adding a default TEST_ENV in the module. -- Antoine
Re: Adding TEST_ENV/UTF-8 to various ports
On 2019/03/15 11:25, Antoine Jacoutot wrote: > On Fri, Mar 15, 2019 at 09:48:39AM +, Stuart Henderson wrote: > > I'd really like if this could be factored into the module so we can get rid > > of copies all over the tree.. perhaps behind a MODPY_PYTEST=Yes and using > > something like MODPY_PYTEST_FLAGS for those that need more than just -m > > pytest? > > Why don't we add it by default for lang/python module ports? > I mean, would it hurt anything? We already have a different default test target for python that uses setup.py.. It's annoying, there are about 3 common methods.
Re: Adding TEST_ENV/UTF-8 to various ports
On Fri, Mar 15, 2019 at 09:48:39AM +, Stuart Henderson wrote:
> I'd really like if this could be factored into the module so we can get rid
> of copies all over the tree.. perhaps behind a MODPY_PYTEST=Yes and using
> something like MODPY_PYTEST_FLAGS for those that need more than just -m
> pytest?
Why don't we add it by default for lang/python module ports?
I mean, would it hurt anything?
> --
> Sent from a phone, apologies for poor formatting.
>
> On 15 March 2019 05:38:44 Kurt Mosiejczuk wrote:
>
> > After sending my diff for productivity/khal that added UTF-8 via a
> > new TEST_DEPENDS and modified do-test target I decided I would look
> > throughout the ports tree rather than doing it as I stumbled across it
> > in individual ports as I worked.
> >
> >
> > I went through all the python ports I found with a do-test that had a
> > pytest construct in them. Many of the ports had no tests that failed
> > because of unicode reasons so I skipped those. The ones that had
> > failures because of unicode complaints about ascii versus UTF-8 I added
> > a TEST_ENV and modified the do-test target. I verified that doing this
> > improved test results in each case.
> >
> >
> > From there I found python ports that already had UTF-8 factored in but
> > did so via a direct inclusion in the test invocation. I abstracted those
> > out and brought in the full SETENV, MAKE_ENV, and TEST_ENV variables
> > in the invocation. Each of those I tested again and the tests came out
> > identically.
> >
> >
> > The latter part is more trying to normalize python ports doing this to a
> > common construct. I do have a version that skips those and just does the
> > ports where it improves test results if that is preferred.
> >
> >
> > Whereas none of this work touches things outside of tests, I did not
> > bump the revision on any of these.
> >
> >
> > cc'ing shadchin@ sebastia@ and remi@ since one or more of the ports this
> > touches are theirs.
> >
> >
> > --Kurt
> >
> >
> > Index: devel/py-cairocffi/Makefile
> > ===
> > RCS file: /cvs/ports/devel/py-cairocffi/Makefile,v
> > retrieving revision 1.1.1.1
> > diff -u -p -r1.1.1.1 Makefile
> > --- devel/py-cairocffi/Makefile 19 Feb 2017 19:11:58 - 1.1.1.1
> > +++ devel/py-cairocffi/Makefile 15 Mar 2019 04:43:01 -
> > @@ -27,7 +27,10 @@ MODPY_SETUPTOOLS = Yes
> > FLAVORS = python3
> > FLAVOR ?=
> >
> > +TEST_ENV += LC_CTYPE=en_US.UTF-8
> > +
> > do-test:
> > - cd ${WRKSRC}/lib && LC_CTYPE=en_US.UTF-8 ${MODPY_BIN} -m pytest cairocffi
> > + cd ${WRKSRC}/lib && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
> > + ${MODPY_BIN} -m pytest cairocffi
> >
> > .include
> > Index: devel/py-click/Makefile
> > ===
> > RCS file: /cvs/ports/devel/py-click/Makefile,v
> > retrieving revision 1.5
> > diff -u -p -r1.5 Makefile
> > --- devel/py-click/Makefile 15 Apr 2017 17:25:39 - 1.5
> > +++ devel/py-click/Makefile 15 Mar 2019 04:43:01 -
> > @@ -25,8 +25,10 @@ FLAVORS= python3
> > FLAVOR?=
> >
> > TEST_DEPENDS= devel/py-test${MODPY_FLAVOR}
> > +TEST_ENV += LC_CTYPE=C.UTF-8
> >
> > do-test:
> > - @cd ${WRKSRC} && ${MODPY_BIN} -m pytest
> > + @cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
> > + ${MODPY_BIN} -m pytest
> >
> > .include
> > Index: devel/py-lazy-object-proxy/Makefile
> > ===
> > RCS file: /cvs/ports/devel/py-lazy-object-proxy/Makefile,v
> > retrieving revision 1.3
> > diff -u -p -r1.3 Makefile
> > --- devel/py-lazy-object-proxy/Makefile 11 May 2017 18:45:11 - 1.3
> > +++ devel/py-lazy-object-proxy/Makefile 15 Mar 2019 04:43:02 -
> > @@ -27,7 +27,10 @@ TEST_DEPENDS = devel/py-test${MODPY_FLA
> > FLAVORS = python3
> > FLAVOR ?=
> >
> > +TEST_ENV += LC_CTYPE=C.UTF-8
> > +
> > do-test:
> > - @cd ${WRKSRC} && LC_CTYPE=C.UTF-8 ${MODPY_BIN} -m pytest --ignore=src
> > + @cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
> > + ${MODPY_BIN} -m pytest --ignore=src
> >
> > .include
> > Index: devel/py-nbconvert/Makefile
> > ===
> > RCS file: /cvs/ports/devel/py-nbconvert/Makefile,v
> > retrieving revision 1.6
> > diff -u -p -r1.6 Makefile
> > --- devel/py-nbconvert/Makefile 25 May 2017 11:45:25 - 1.6
> > +++ devel/py-nbconvert/Makefile 15 Mar 2019 04:43:02 -
> > @@ -45,7 +45,10 @@ TEST_DEPENDS = ${BASE_PKGPATH}=${MODPY_
> > post-install:
> > mv ${PREFIX}/bin/jupyter-nbconvert{,${MODPY_BIN_SUFFIX}}
> >
> > +TEST_ENV += LC_CTYPE=en_US.UTF-8
> > +
> > do-test:
> > - cd ${WRKSRC} && LC_CTYPE=en_US.UTF-8 ${MODPY_BIN} -m pytest nbconvert
> > + cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
> > + ${MODPY_BIN} -m pytest nbconvert
> >
> > .include
> > Index: devel/py-nbformat/Makefile
> > ===
> > RCS file:
Re: Adding TEST_ENV/UTF-8 to various ports
I'd really like if this could be factored into the module so we can get rid
of copies all over the tree.. perhaps behind a MODPY_PYTEST=Yes and using
something like MODPY_PYTEST_FLAGS for those that need more than just -m pytest?
--
Sent from a phone, apologies for poor formatting.
On 15 March 2019 05:38:44 Kurt Mosiejczuk wrote:
After sending my diff for productivity/khal that added UTF-8 via a
new TEST_DEPENDS and modified do-test target I decided I would look
throughout the ports tree rather than doing it as I stumbled across it
in individual ports as I worked.
I went through all the python ports I found with a do-test that had a
pytest construct in them. Many of the ports had no tests that failed
because of unicode reasons so I skipped those. The ones that had
failures because of unicode complaints about ascii versus UTF-8 I added
a TEST_ENV and modified the do-test target. I verified that doing this
improved test results in each case.
From there I found python ports that already had UTF-8 factored in but
did so via a direct inclusion in the test invocation. I abstracted those
out and brought in the full SETENV, MAKE_ENV, and TEST_ENV variables
in the invocation. Each of those I tested again and the tests came out
identically.
The latter part is more trying to normalize python ports doing this to a
common construct. I do have a version that skips those and just does the
ports where it improves test results if that is preferred.
Whereas none of this work touches things outside of tests, I did not
bump the revision on any of these.
cc'ing shadchin@ sebastia@ and remi@ since one or more of the ports this
touches are theirs.
--Kurt
Index: devel/py-cairocffi/Makefile
===
RCS file: /cvs/ports/devel/py-cairocffi/Makefile,v
retrieving revision 1.1.1.1
diff -u -p -r1.1.1.1 Makefile
--- devel/py-cairocffi/Makefile 19 Feb 2017 19:11:58 - 1.1.1.1
+++ devel/py-cairocffi/Makefile 15 Mar 2019 04:43:01 -
@@ -27,7 +27,10 @@ MODPY_SETUPTOOLS = Yes
FLAVORS = python3
FLAVOR ?=
+TEST_ENV += LC_CTYPE=en_US.UTF-8
+
do-test:
- cd ${WRKSRC}/lib && LC_CTYPE=en_US.UTF-8 ${MODPY_BIN} -m pytest cairocffi
+ cd ${WRKSRC}/lib && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
+ ${MODPY_BIN} -m pytest cairocffi
.include
Index: devel/py-click/Makefile
===
RCS file: /cvs/ports/devel/py-click/Makefile,v
retrieving revision 1.5
diff -u -p -r1.5 Makefile
--- devel/py-click/Makefile 15 Apr 2017 17:25:39 - 1.5
+++ devel/py-click/Makefile 15 Mar 2019 04:43:01 -
@@ -25,8 +25,10 @@ FLAVORS= python3
FLAVOR?=
TEST_DEPENDS= devel/py-test${MODPY_FLAVOR}
+TEST_ENV += LC_CTYPE=C.UTF-8
do-test:
- @cd ${WRKSRC} && ${MODPY_BIN} -m pytest
+ @cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
+ ${MODPY_BIN} -m pytest
.include
Index: devel/py-lazy-object-proxy/Makefile
===
RCS file: /cvs/ports/devel/py-lazy-object-proxy/Makefile,v
retrieving revision 1.3
diff -u -p -r1.3 Makefile
--- devel/py-lazy-object-proxy/Makefile 11 May 2017 18:45:11 - 1.3
+++ devel/py-lazy-object-proxy/Makefile 15 Mar 2019 04:43:02 -
@@ -27,7 +27,10 @@ TEST_DEPENDS = devel/py-test${MODPY_FLA
FLAVORS = python3
FLAVOR ?=
+TEST_ENV += LC_CTYPE=C.UTF-8
+
do-test:
- @cd ${WRKSRC} && LC_CTYPE=C.UTF-8 ${MODPY_BIN} -m pytest --ignore=src
+ @cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
+ ${MODPY_BIN} -m pytest --ignore=src
.include
Index: devel/py-nbconvert/Makefile
===
RCS file: /cvs/ports/devel/py-nbconvert/Makefile,v
retrieving revision 1.6
diff -u -p -r1.6 Makefile
--- devel/py-nbconvert/Makefile 25 May 2017 11:45:25 - 1.6
+++ devel/py-nbconvert/Makefile 15 Mar 2019 04:43:02 -
@@ -45,7 +45,10 @@ TEST_DEPENDS = ${BASE_PKGPATH}=${MODPY_
post-install:
mv ${PREFIX}/bin/jupyter-nbconvert{,${MODPY_BIN_SUFFIX}}
+TEST_ENV += LC_CTYPE=en_US.UTF-8
+
do-test:
- cd ${WRKSRC} && LC_CTYPE=en_US.UTF-8 ${MODPY_BIN} -m pytest nbconvert
+ cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
+ ${MODPY_BIN} -m pytest nbconvert
.include
Index: devel/py-nbformat/Makefile
===
RCS file: /cvs/ports/devel/py-nbformat/Makefile,v
retrieving revision 1.4
diff -u -p -r1.4 Makefile
--- devel/py-nbformat/Makefile 23 Feb 2017 11:52:00 - 1.4
+++ devel/py-nbformat/Makefile 15 Mar 2019 04:43:02 -
@@ -29,10 +29,13 @@ MODPY_PI = Yes
FLAVORS = python3
FLAVOR ?=
+TEST_ENV += LC_CTYPE=en_US.UTF-8
+
post-install:
mv ${PREFIX}/bin/jupyter-trust ${PREFIX}/bin/jupyter-trust${MODPY_BIN_SUFFIX}
do-test:
- cd ${WRKSRC} && LC_CTYPE=en_US.UTF-8 ${MODPY_BIN} -m pytest nbformat
+ cd ${WRKSRC} && exec ${SETENV} ${MAKE_ENV} ${TEST_ENV} \
+ ${MODPY_BIN} -m pytest nbformat
.include
Index: devel/py-path.py/Makefile
