Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
> On Wed, 13 Sep 2023, Eli Schwartz wrote:
>> That's a rather bold statement. I can imagine a number of possible
>> failures, e.g. no space left on device, quota exceeded, or a low-level
>> I/O error of the filesystem. Also fork or exec of the cat command could
>> fail (e.g. out of memory).
>>
>> While either of these may be unlikely here, best practice is to check
>> for errors of _all_ external commands.
> The implementation of `die` performs a fork+exec of the sed command,
> invokes eerror (many times!) which forks to pipe, invokes $() which
> forks, and could behave abnormally due to out of memory. It is not safe
> to treat `|| die` as error recovery for an out of memory condition.
> The implementation of `die` performs multiple writes to files inside of
> ${PORTAGE_BUILDDIR}, and could behave abnormally due to write failures.
> It is not safe to treat `|| die` as error recovery for a write failure
> of the ${PORTAGE_BUILDDIR} drive (whatever the reason for that write
> failure).
An eclass must not rely on implementation details of any specific
package manager.
"die [...] aborts the build process."
https://projects.gentoo.org/pms/8/pms.html#x1-12600012.3.6
So please report a Portage (or other package manager) bug if you can
reproduce a condition where die doesn't abort the build process.
> (And the `|| die` is added to the next revision of this patch either way.)
Thanks.
Ulrich
signature.asc
Description: PGP signature
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
On 9/13/23 9:10 AM, Michael Orlitzky wrote: > On Tue, 2023-09-12 at 22:52 -0400, Eli Schwartz wrote: >> >> Is portage generally expected to successfully complete (including >> internal metadata write stages) when its workdir drive runs out of space >> partway through? >> > > No, but I think what everyone else is forgetting to mention is that if > it's going to fail, we want it to fail as soon as possible, i.e. as > close to the thing that actually went wrong. If we proceed past ENOSPC > or whatever, we could get five or six lines further in the ebuild, and > then some other command will fail but possibly with a crazy unrelated > error message. The consequences of this particular command failing are that: - setuptools will still build, but without spawning a ThreadPoolExecutor to distribute jobs; it builds single threaded - setuptools will still build, but using its default build directory instead of an alternative one; setuptools itself *cannot* fail as a result, but other external commands hardcoding the location of setuptools internal files could fail to find those files (It will still fail when setuptools reports its own ENOSPC, but this is not a crazy unrelated error message.) (Unless portage expects to successfully complete where possible if the workdir drive runs out of space partway through, then quickly has more space freed up for it as soon as monitoring warnings are triggered and no actually-fatal ebuild errors occurred during the window of ENOSPC.) -- Eli Schwartz
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
On 9/13/23 5:27 AM, Ulrich Mueller wrote:
>> On Wed, 13 Sep 2023, Eli Schwartz wrote:
>
>>> "|| die" should also be added for the cat command.
>
>> Redirecting output to a file in a directory you have just guaranteed
>> to exist cannot fail.
>
> That's a rather bold statement. I can imagine a number of possible
> failures, e.g. no space left on device, quota exceeded, or a low-level
> I/O error of the filesystem. Also fork or exec of the cat command could
> fail (e.g. out of memory).
>
> While either of these may be unlikely here, best practice is to check
> for errors of _all_ external commands.
The implementation of `die` performs a fork+exec of the sed command,
invokes eerror (many times!) which forks to pipe, invokes $() which
forks, and could behave abnormally due to out of memory. It is not safe
to treat `|| die` as error recovery for an out of memory condition.
The implementation of `die` performs multiple writes to files inside of
${PORTAGE_BUILDDIR}, and could behave abnormally due to write failures.
It is not safe to treat `|| die` as error recovery for a write failure
of the ${PORTAGE_BUILDDIR} drive (whatever the reason for that write
failure).
Regarding:
- no space left on device
- quota exceeded
- low-level I/O error of the filesystem
this isn't "a number of possible failures" :) it is the same failure but
elicited by different prompts. Regarding this, I have already commented...
(And the `|| die` is added to the next revision of this patch either way.)
--
Eli Schwartz
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
On Tue, 2023-09-12 at 22:52 -0400, Eli Schwartz wrote: > > Is portage generally expected to successfully complete (including > internal metadata write stages) when its workdir drive runs out of space > partway through? > No, but I think what everyone else is forgetting to mention is that if it's going to fail, we want it to fail as soon as possible, i.e. as close to the thing that actually went wrong. If we proceed past ENOSPC or whatever, we could get five or six lines further in the ebuild, and then some other command will fail but possibly with a crazy unrelated error message.
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
> On Wed, 13 Sep 2023, Eli Schwartz wrote: >> "|| die" should also be added for the cat command. > Redirecting output to a file in a directory you have just guaranteed > to exist cannot fail. That's a rather bold statement. I can imagine a number of possible failures, e.g. no space left on device, quota exceeded, or a low-level I/O error of the filesystem. Also fork or exec of the cat command could fail (e.g. out of memory). While either of these may be unlikely here, best practice is to check for errors of _all_ external commands.
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
13.09.2023 03:52, Eli Schwartz пишет:
On 9/12/23 10:26 PM, Michał Górny wrote:
On Tue, 2023-09-12 at 22:07 -0400, Eli Schwartz wrote:
On 9/12/23 3:56 PM, Ulrich Mueller wrote:
On Tue, 12 Sep 2023, Eli Schwartz wrote:
+ mkdir -p "${BUILD_DIR}" || die
+ local -x
DIST_EXTRA_CONFIG="${BUILD_DIR}/extra-setup.cfg"
+ cat > "${DIST_EXTRA_CONFIG}" <<-EOF
+ [build]
+ build_base = ${BUILD_DIR}/build
+
+ [build_ext]
+ parallel = ${jobs}
+ EOF
"|| die" should also be added for the cat command.
Redirecting output to a file in a directory you have just guaranteed to
exist cannot fail.
Eh, you make me prove you wrong:
# cat > dupa <<-EOF
blahblah
EOF
cat: write error: No space left on device
ಠ_ಠ
Is portage generally expected to successfully complete (including
internal metadata write stages) when its workdir drive runs out of space
partway through?
This is a race with the user - the user could delete some files from
disk just in time for metadata to successfully be written
--
Best regards,
Alexey "DarthGandalf" Sokolov
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
Eli Schwartz writes:
> On 9/12/23 10:26 PM, Michał Górny wrote:
>> On Tue, 2023-09-12 at 22:07 -0400, Eli Schwartz wrote:
>>> On 9/12/23 3:56 PM, Ulrich Mueller wrote:
> On Tue, 12 Sep 2023, Eli Schwartz wrote:
> + mkdir -p "${BUILD_DIR}" || die
> + local -x
> DIST_EXTRA_CONFIG="${BUILD_DIR}/extra-setup.cfg"
> + cat > "${DIST_EXTRA_CONFIG}" <<-EOF
> + [build]
> + build_base = ${BUILD_DIR}/build
> +
> + [build_ext]
> + parallel = ${jobs}
> + EOF
"|| die" should also be added for the cat command.
>>>
>>>
>>> Redirecting output to a file in a directory you have just guaranteed to
>>> exist cannot fail.
>>
>> Eh, you make me prove you wrong:
>>
>> # cat > dupa <<-EOF
>> blahblah
>>> EOF
>> cat: write error: No space left on device
>
>
> ಠ_ಠ
>
> Is portage generally expected to successfully complete (including
> internal metadata write stages) when its workdir drive runs out of space
> partway through?
oh you
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
On 9/12/23 10:26 PM, Michał Górny wrote:
> On Tue, 2023-09-12 at 22:07 -0400, Eli Schwartz wrote:
>> On 9/12/23 3:56 PM, Ulrich Mueller wrote:
On Tue, 12 Sep 2023, Eli Schwartz wrote:
>>>
+ mkdir -p "${BUILD_DIR}" || die
+ local -x
DIST_EXTRA_CONFIG="${BUILD_DIR}/extra-setup.cfg"
+ cat > "${DIST_EXTRA_CONFIG}" <<-EOF
+ [build]
+ build_base = ${BUILD_DIR}/build
+
+ [build_ext]
+ parallel = ${jobs}
+ EOF
>>>
>>> "|| die" should also be added for the cat command.
>>
>>
>> Redirecting output to a file in a directory you have just guaranteed to
>> exist cannot fail.
>
> Eh, you make me prove you wrong:
>
> # cat > dupa <<-EOF
> blahblah
>> EOF
> cat: write error: No space left on device
ಠ_ಠ
Is portage generally expected to successfully complete (including
internal metadata write stages) when its workdir drive runs out of space
partway through?
--
Eli Schwartz
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
On Tue, 2023-09-12 at 22:07 -0400, Eli Schwartz wrote:
> On 9/12/23 3:56 PM, Ulrich Mueller wrote:
> > > > > > > On Tue, 12 Sep 2023, Eli Schwartz wrote:
> >
> > > + mkdir -p "${BUILD_DIR}" || die
> > > + local -x
> > > DIST_EXTRA_CONFIG="${BUILD_DIR}/extra-setup.cfg"
> > > + cat > "${DIST_EXTRA_CONFIG}" <<-EOF
> > > + [build]
> > > + build_base = ${BUILD_DIR}/build
> > > +
> > > + [build_ext]
> > > + parallel = ${jobs}
> > > + EOF
> >
> > "|| die" should also be added for the cat command.
>
>
> Redirecting output to a file in a directory you have just guaranteed to
> exist cannot fail.
Eh, you make me prove you wrong:
# cat > dupa <<-EOF
blahblah
> EOF
cat: write error: No space left on device
--
Best regards,
Michał Górny
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
On 9/12/23 3:56 PM, Ulrich Mueller wrote:
>> On Tue, 12 Sep 2023, Eli Schwartz wrote:
>
>> +mkdir -p "${BUILD_DIR}" || die
>> +local -x
>> DIST_EXTRA_CONFIG="${BUILD_DIR}/extra-setup.cfg"
>> +cat > "${DIST_EXTRA_CONFIG}" <<-EOF
>> +[build]
>> +build_base = ${BUILD_DIR}/build
>> +
>> +[build_ext]
>> +parallel = ${jobs}
>> +EOF
>
> "|| die" should also be added for the cat command.
Redirecting output to a file in a directory you have just guaranteed to
exist cannot fail. (mkdir -p will return nonzero and die, if it exits
without having guaranteed its target is a directory on disk. There is
one loophole, and that is if its target already existed beforehand, but
the mkdir -p process did not have write permissions for it; mkdir will
not check permissions if it detects it doesn't have to do work. This
should not be the case inside the workdir, or something has gone
significantly wrong beforehand.)
That being said, as a style guide concern I can see why being cautious
is generally desirable, as it's much easier to review code that uses it
when it isn't strictly necessary than code that doesn't use it when it
turns out to be necessary.
I'm not very used to this yet :) so I blindly assumed while writing it
that it wasn't necessary to do some additional typing and add this for
call sites that shouldn't be able to fail. I will add this in the next
revision.
--
Eli Schwartz
Re: [gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
> On Tue, 12 Sep 2023, Eli Schwartz wrote:
> + mkdir -p "${BUILD_DIR}" || die
> + local -x
> DIST_EXTRA_CONFIG="${BUILD_DIR}/extra-setup.cfg"
> + cat > "${DIST_EXTRA_CONFIG}" <<-EOF
> + [build]
> + build_base = ${BUILD_DIR}/build
> +
> + [build_ext]
> + parallel = ${jobs}
> + EOF
"|| die" should also be added for the cat command.
[gentoo-dev] [PATCH 1/2] distutils-r1.eclass: teach setuptools to respect (some) build options
Previously, setup.py was handled by:
- manually passing makejobs, with a heuristic to guess whether it was a
time saver to do so.
- rm -rf'ing the build directory in between python versions to prevent
cross-version contamination
This is because in PEP 517 mode, it doesn't accept build options
specific to a setuptools phase. So a crude hack is to just build_ext
twice, once explicitly and once internally as part of bdist_wheel, and
pray that in the latter case it detects that there's nothing to do.
Unfortunately, sometimes build_ext does NOT detect that there is nothing
to do -- e.g. for codegen tools such as mypyc, that produce *.c files
which are different every time you try building. As for build
directories, those were given up on as hopeless.
There's a better hack which is to set a magic environment variable for a
setup.cfg file which is parsed additionally to the one provided by the
project. It can contain additional settings, such as the build-base and
parallelism, which means that bdist_wheel intrinsically builds
extensions in parallel the only time it is called. And we can set the
output directory for all build artifacts to outside of the source tree,
so it is no longer necessary to delete them (which among other things,
makes debugging difficult).
This is similar to .pydistutils.cfg, but is processed later and can be
in arbitrary locations. Since we store it in the per-impl build
directory we don't need to wipe it after using it to avoid leakage.
Signed-off-by: Eli Schwartz
---
eclass/distutils-r1.eclass | 48 --
1 file changed, 10 insertions(+), 38 deletions(-)
diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 91de144e1110..dd197a5f0693 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1461,12 +1461,6 @@ distutils_pep517_install() {
[[ -n ${wheel} ]] || die "No wheel name returned"
distutils_wheel_install "${root}" "${WHEEL_BUILD_DIR}/${wheel}"
-
- # clean the build tree; otherwise we may end up with PyPy3
- # extensions duplicated into CPython dists
- if [[ ${DISTUTILS_USE_PEP517:-setuptools} == setuptools ]]; then
- rm -rf build || die
- fi
}
# @FUNCTION: distutils-r1_python_compile
@@ -1478,9 +1472,6 @@ distutils_pep517_install() {
#
# If DISTUTILS_USE_PEP517 is set to any other value, builds a wheel
# using the PEP517 backend and installs it into ${BUILD_DIR}/install.
-# May additionally call build_ext prior to that when using setuptools
-# and the eclass detects a potential benefit from parallel extension
-# builds.
#
# In legacy mode, runs 'esetup.py build'. Any parameters passed to this
# function will be appended to setup.py invocation, i.e. passed
@@ -1495,40 +1486,21 @@ distutils-r1_python_compile() {
# call setup.py build when using setuptools (either via
PEP517
# or in legacy mode)
- if [[ ${DISTUTILS_USE_PEP517} ]]; then
- if [[ -d build ]]; then
- eqawarn "A 'build' directory exists
already. Artifacts from this directory may"
- eqawarn "be picked up by setuptools
when building for another interpreter."
- eqawarn "Please remove this directory
prior to building."
- fi
- else
- _distutils-r1_copy_egg_info
- fi
-
# distutils is parallel-capable since py3.5
local jobs=$(makeopts_jobs "${MAKEOPTS} ${*}")
if [[ ${DISTUTILS_USE_PEP517} ]]; then
- # issue build_ext only if it looks like we have
at least
- # two source files to build; setuptools is
expensive
- # to start and parallel builds can only benefit
us if we're
- # compiling at least two files
- #
- # see extension.py for list of suffixes
- # .pyx is added for Cython
- #
- # esetup.py does not respect SYSROOT, so skip
it there
- if [[ -z ${SYSROOT} && ${DISTUTILS_EXT} && 1
-ne ${jobs}
- && 2 -eq $(
- find '(' -name '*.c' -o -name
'*.cc' -o -name '*.cpp' \
- -o -name '*.cxx' -o
-name '*.c++' -o -name '*.m' \
- -o -name '*.mm' -o
-name '*.pyx' ')' -printf '\n' |
- head -n 2 | wc -l
- )
-
