Re: [gentoo-dev] Last rites: */* More Py2 only items
On Sun, Jun 28, 2020 at 5:35 PM Aaron Bauman wrote: > # Aaron Bauman (2020-06-28) > # More Py2 only stuff. Plz see -dev ML for discussions > # Remove bindings, port to Py3, etc > # Removal in 30 days > app-arch/deltarpm > app-crypt/virtualsmartcard > app-text/duali > app-text/mftrace > app-text/queequeg > app-text/referencer > dev-libs/libmacaroons > dev-libs/tut > dev-python/elib-intl > dev-python/eunuchs > dev-python/medusa > dev-python/misaka > dev-python/python-iwscan > dev-util/confix > dev-util/qmtest > dev-util/unrpyc > games-engines/gemrb > media-sound/lilycomp > media-video/tovid > net-dns/maradns > net-irc/irker > net-mail/archivemail > net-mail/getmai > net-nds/nsscache > Infra uses this, we will have to look into bumping it. -A > net-wireless/airpwn > sci-chemistry/bkchem > sci-chemistry/propka > sci-chemistry/pymol-plugins-bni-tools > sci-chemistry/pymol-plugins-emovie > sci-chemistry/viewmol > sci-libs/chemkit > sys-apps/x86info > www-misc/surl > x11-wm/plwm > > -- > Cheers, > Aaron >
[gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2020-06-28 23:59 UTC
The attached list notes all of the packages that were added or removed from the tree, for the week ending 2020-06-28 23:59 UTC. Removals: app-metrics/buildbot-prometheus 20200627-00:40 bman bc8bdd812f7 dev-db/clickhouse 20200627-00:49 bman 72ae4614583 net-fs/ncpfs20200627-00:45 bman d427fe8c7ee net-misc/charm 20200627-00:35 bman 5b83079c4f1 net-misc/ipx-utils 20200627-00:45 bman d427fe8c7ee sys-cluster/polysh 20200627-00:35 bman 5b83079c4f1 Additions: acct-group/cvmfs20200626-13:05 amadio98e82738a8c acct-group/sabnzbd 20200628-12:36 whissi507799c2fe1 acct-user/cvmfs 20200626-13:07 amadio196600334c4 acct-user/sabnzbd 20200628-12:37 whissi78fc1154672 app-text/vgrep 20200628-22:58 whissi24d5a90e18b dev-libs/libclangformat-ide 20200519-01:15 zx2c4 dd9332b486f dev-ml/stdlib-shims 20200612-03:18 juippis 4f065399f83 dev-python/latexcodec 20200502-12:22 juippis 669176a7eef dev-python/pybtex 20200602-10:16 juippis eb876389142 dev-qt/qtquicktimeline 20200511-01:35 zx2c4 d4797905da3 dev-util/chroot-wrapper 20200627-19:47 chutzpah f96df9fce30 games-misc/usolitaire 20200625-05:26 monsieurp 1a6f528d0ba games-util/acc 20200622-20:19 chewi e2b9f0ceb1d media-gfx/netpaint 20200625-06:17 monsieurp 9ec66d490fd net-irc/irccloud-desktop-bin20200625-00:40 zx2c4 0cfb3563464 sci-mathematics/mathematica 20200624-14:10 dilfridge c288c4436e2 sys-cluster/k9scli 20200625-19:51 williamh cb6b0565d7a www-apache/mod_maxminddb20200430-21:32 whissi1df35586c28 -- Robin Hugh Johnson Gentoo Linux Developer E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 Removed Packages: dev-db/clickhouse,removed,bman,20200627-00:49,72ae4614583 net-fs/ncpfs,removed,bman,20200627-00:45,d427fe8c7ee net-misc/ipx-utils,removed,bman,20200627-00:45,d427fe8c7ee app-metrics/buildbot-prometheus,removed,bman,20200627-00:40,bc8bdd812f7 net-misc/charm,removed,bman,20200627-00:35,5b83079c4f1 sys-cluster/polysh,removed,bman,20200627-00:35,5b83079c4f1 Added Packages: app-text/vgrep,added,whissi,20200628-22:58,24d5a90e18b acct-user/sabnzbd,added,whissi,20200628-12:37,78fc1154672 acct-group/sabnzbd,added,whissi,20200628-12:36,507799c2fe1 dev-python/pybtex,added,juippis,20200602-10:16,eb876389142 dev-python/latexcodec,added,juippis,20200502-12:22,669176a7eef dev-util/chroot-wrapper,added,chutzpah,20200627-19:47,f96df9fce30 www-apache/mod_maxminddb,added,whissi,20200430-21:32,1df35586c28 acct-user/cvmfs,added,amadio,20200626-13:07,196600334c4 acct-group/cvmfs,added,amadio,20200626-13:05,98e82738a8c sys-cluster/k9scli,added,williamh,20200625-19:51,cb6b0565d7a dev-libs/libclangformat-ide,added,zx2c4,20200519-01:15,dd9332b486f dev-qt/qtquicktimeline,added,zx2c4,20200511-01:35,d4797905da3 media-gfx/netpaint,added,monsieurp,20200625-06:17,9ec66d490fd games-misc/usolitaire,added,monsieurp,20200625-05:26,1a6f528d0ba net-irc/irccloud-desktop-bin,added,zx2c4,20200625-00:40,0cfb3563464 sci-mathematics/mathematica,added,dilfridge,20200624-14:10,c288c4436e2 dev-ml/stdlib-shims,added,juippis,20200612-03:18,4f065399f83 games-util/acc,added,chewi,20200622-20:19,e2b9f0ceb1d Done.
Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function
On Sat, Jun 27, 2020, at 1:34 AM, Chun-Yu Shei wrote:
> According to cProfile, catpkgsplit is called up to 1-5.5 million times
> during "emerge -uDvpU --with-bdeps=y @world". Adding a dict to cache its
> results reduces the time for this command from 43.53 -> 41.53 seconds --
> a 4.8% speedup.
> ---
> lib/portage/versions.py | 7 +++
> 1 file changed, 7 insertions(+)
>
> diff --git a/lib/portage/versions.py b/lib/portage/versions.py
> index 0c21373cc..ffec316ce 100644
> --- a/lib/portage/versions.py
> +++ b/lib/portage/versions.py
> @@ -312,6 +312,7 @@ def _pkgsplit(mypkg, eapi=None):
>
> _cat_re = re.compile('^%s$' % _cat, re.UNICODE)
> _missing_cat = 'null'
> +_catpkgsplit_cache = {}
>
> def catpkgsplit(mydata, silent=1, eapi=None):
> """
> @@ -331,6 +332,11 @@ def catpkgsplit(mydata, silent=1, eapi=None):
> return mydata.cpv_split
> except AttributeError:
> pass
> +
> + cache_entry = _catpkgsplit_cache.get(mydata)
> + if cache_entry is not None:
> + return cache_entry
> +
> mysplit = mydata.split('/', 1)
> p_split = None
> if len(mysplit) == 1:
> @@ -343,6 +349,7 @@ def catpkgsplit(mydata, silent=1, eapi=None):
> if not p_split:
> return None
> retval = (cat, p_split[0], p_split[1], p_split[2])
> + _catpkgsplit_cache[mydata] = retval
> return retval
>
> class _pkg_str(_unicode):
> --
> 2.27.0.212.ge8ba1cc988-goog
>
There are libraries that provide decorators, etc, for caching and memoization.
Have you evaluated any of those? One is available in the standard library:
https://docs.python.org/dev/library/functools.html#functools.lru_cache
I comment as this would increase code clarity.
[gentoo-dev] Last rites: */* More Py2 only items
# Aaron Bauman (2020-06-28) # More Py2 only stuff. Plz see -dev ML for discussions # Remove bindings, port to Py3, etc # Removal in 30 days app-arch/deltarpm app-crypt/virtualsmartcard app-text/duali app-text/mftrace app-text/queequeg app-text/referencer dev-libs/libmacaroons dev-libs/tut dev-python/elib-intl dev-python/eunuchs dev-python/medusa dev-python/misaka dev-python/python-iwscan dev-util/confix dev-util/qmtest dev-util/unrpyc games-engines/gemrb media-sound/lilycomp media-video/tovid net-dns/maradns net-irc/irker net-mail/archivemail net-mail/getmai net-nds/nsscache net-wireless/airpwn sci-chemistry/bkchem sci-chemistry/propka sci-chemistry/pymol-plugins-bni-tools sci-chemistry/pymol-plugins-emovie sci-chemistry/viewmol sci-libs/chemkit sys-apps/x86info www-misc/surl x11-wm/plwm -- Cheers, Aaron signature.asc Description: PGP signature
Re: [gentoo-dev] Last rites: */* More Py2 only items
On Sun, Jun 28, 2020 at 08:35:02PM -0400, Aaron Bauman wrote: > # Aaron Bauman (2020-06-28) > # More Py2 only stuff. Plz see -dev ML for discussions > # Remove bindings, port to Py3, etc > # Removal in 30 days > app-arch/deltarpm > app-crypt/virtualsmartcard > app-text/duali Add app-text/duali-data which is an rdep of app-text/duali -- Cheers, Aaron signature.asc Description: PGP signature
[gentoo-dev] RFC: Standard build environment variables
As many of you probably know, ago@ has been expanding the scope of our CFLAGS/CC support to include some other common build variables: * CC * CXX * AR * CPP * NM * RANLIB * AS * LD Some of those are POSIX standards[0], * CC * AR Others are de-facto GNU make standards[1], * CXX * CPP * AS and a few are de-facto GNU libtool standards[2]: * NM * RANLIB * LD If we expect them all to work properly in Gentoo, we have to agree on what they mean, and thus how they should be injected into build systems. For example, we had a problem with sci-mathematics/pari, whose upstream is using the LD environment variable for something other than what GNU libtool uses it for. With LD set to something libtooly in the environment, the pari build fails. We can solve that by unsetting LD in the ebuild, but for that to be The Right Thing To Do, we should be expecting LD to contain something libtooly, and thus something inappropriate to be passed to the pari build. To avoid these issues, I suggest creating a list of "Gentoo environment variables" in the devmanual with descriptions of how they should be used and pointers to the references (for why we chose that meaning). That way a user can export LD, for example, and know that it will be used how he thinks it will be used. [0] https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html [1] https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html [2] https://www.gnu.org/software/libtool/manual/libtool.html
Re: [gentoo-dev] RFC: Standard build environment variables
On Sun, 28 Jun 2020 08:18:23 -0400 Michael Orlitzky wrote: > With LD set to something libtooly in the > environment, the pari build fails. We can solve that by unsetting LD in > the ebuild, but for that to be The Right Thing To Do, we should be > expecting LD to contain something libtooly, and thus something > inappropriate to be passed to the pari build. > > To avoid these issues, I suggest creating a list of "Gentoo environment > variables" in the devmanual with descriptions of how they should be used > and pointers to the references (for why we chose that meaning). That way > a user can export LD, for example, and know that it will be used how he > thinks it will be used. That LD problem is not unique to Pari, ftr. Its a problem in *all* Perl things that involve C parts. Because Perl code expects "LD" to be a "CCLD", and passes flags a CCLD would expect, which LD can't handle. pgpasaW8H6QqH.pgp Description: OpenPGP digital signature
[gentoo-dev] Last rites: app-misc/scope
# Mikle Kolyada (2020-06-28) # Obsolete package. # Does not build. # Dead upstream and only gentoo ships it. # Removal in 30 days. app-misc/scope signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC: Standard build environment variables
On Sun, Jun 28, 2020 at 8:18 AM Michael Orlitzky wrote: > > As many of you probably know, ago@ has been expanding the scope of our > CFLAGS/CC support to include some other common build variables: > > * CC > * CXX > * AR > * CPP > * NM > * RANLIB > * AS > * LD > > Some of those are POSIX standards[0], > > * CC > * AR > > Others are de-facto GNU make standards[1], > > * CXX > * CPP > * AS > > and a few are de-facto GNU libtool standards[2]: > > * NM > * RANLIB > * LD > > If we expect them all to work properly in Gentoo, we have to agree on > what they mean, and thus how they should be injected into build systems. > For example, we had a problem with sci-mathematics/pari, whose upstream > is using the LD environment variable for something other than what GNU > libtool uses it for. With LD set to something libtooly in the > environment, the pari build fails. We can solve that by unsetting LD in > the ebuild, but for that to be The Right Thing To Do, we should be > expecting LD to contain something libtooly, and thus something > inappropriate to be passed to the pari build. > > To avoid these issues, I suggest creating a list of "Gentoo environment > variables" in the devmanual with descriptions of how they should be used > and pointers to the references (for why we chose that meaning). That way > a user can export LD, for example, and know that it will be used how he > thinks it will be used. Makes sense to me.
[gentoo-dev] Packages up for grabs: app-doc/zeal, x11-terms/pangoterm, maybe dev-ruby/neovim-ruby-client
Hi, Due to the maintainer going AWOL, the following packages now need new maintainer: app-doc/zeal x11-terms/pangoterm Both have a single bug reported. There's also one package that fell back to ruby project as backup: dev-ruby/neovim-ruby-client -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH] ecompress: optimize docompress -x precompressed comparison
Use sort and comm with temporary files in order to compare lists
of docompress -x and precompressed files, since the file lists
can be extremely large. Also strip ${D%/} from paths in order to
reduce length.
Bug: https://bugs.gentoo.org/721516
Suggested-by: Robin H. Johnson
Signed-off-by: Zac Medico
---
bin/ecompress | 29 ++-
.../tests/resolver/ResolverPlayground.py | 1 +
2 files changed, 16 insertions(+), 14 deletions(-)
diff --git a/bin/ecompress b/bin/ecompress
index 60b083834..983a4d1f7 100755
--- a/bin/ecompress
+++ b/bin/ecompress
@@ -19,29 +19,30 @@ while [[ $# -gt 0 ]] ; do
shift
skip_dirs=()
- skip_files=()
+ > "${T}/.ecompress_skip_files" || die
for skip; do
if [[ -d ${ED%/}/${skip#/} ]]; then
skip_dirs+=( "${ED%/}/${skip#/}" )
else
rm -f "${ED%/}/${skip#/}.ecompress" || die
- skip_files+=("${ED%/}/${skip#/}")
+ printf '%s\0' "${EPREFIX}/${skip#/}" >>
"${T}/.ecompress_skip_files"
fi
done
if [[ ${#skip_dirs[@]} -gt 0 ]]; then
- while read -r -d ''; do
- skip_files+=("${REPLY%.ecompress}")
+ while read -r -d '' skip; do
+ skip=${skip%.ecompress}
+ printf '%s\0' "${skip#${D%/}}" >>
"${T}/.ecompress_skip_files"
done < <(find "${skip_dirs[@]}" -name '*.ecompress'
-print0 -delete || die)
fi
- if [[ ${#skip_files[@]} -gt 0 && -s
${T}/.ecompress_had_precompressed ]]; then
- sed_args=()
- for f in "${skip_files[@]}"; do
- sed_args+=("s|^${f}\$||;")
- done
- sed_args+=('/^$/d')
- sed -f - -i "${T}/.ecompress_had_precompressed" <<<
"${sed_args[@]}" || die
+ if [[ -s ${T}/.ecompress_skip_files && -s
${T}/.ecompress_had_precompressed ]]; then
+ # Filter skipped files from
${T}/.ecompress_had_precompressed,
+ # using temporary files since these lists can be
extremely large.
+ LC_COLLATE=C sort -zu "${T}/.ecompress_skip_files" >
"${T}/.ecompress_skip_files_sorted"|| die
+ LC_COLLATE=C sort -zu
"${T}/.ecompress_had_precompressed" >
"${T}/.ecompress_had_precompressed_sorted" || die
+ LC_COLLATE=C comm -z13
"${T}/.ecompress_skip_files_sorted" "${T}/.ecompress_had_precompressed_sorted"
> "${T}/.ecompress_had_precompressed" || die
+ rm -f "${T}/.ecompress_had_precompressed_sorted"
"${T}/.ecompress_skip_files"{,_sorted}
fi
exit 0
@@ -81,7 +82,7 @@ while [[ $# -gt 0 ]] ; do
continue 2
fi
done
- echo "${path}" >>
"${T}"/.ecompress_had_precompressed
+ printf '%s\0' "${path#${D%/}}"
>> "${T}"/.ecompress_had_precompressed || die
;;
esac
@@ -195,8 +196,8 @@ if [[ -s ${T}/.ecompress_had_precompressed ]]; then
eqawarn "(manpages, documentation) when automatic compression is used:"
eqawarn
n=0
- while read -r f; do
- eqawarn " ${f#${D%/}}"
+ while read -r -d '' f; do
+ eqawarn " ${f}"
if [[ $(( n++ )) -eq 10 ]]; then
eqawarn " ..."
break
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
b/lib/portage/tests/resolver/ResolverPlayground.py
index de80a0cc1..ec2e31ae9 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -91,6 +91,7 @@ class ResolverPlayground(object):
"chgrp",
"chmod",
"chown",
+ "comm",
"cp",
"egrep",
"env",
--
2.25.3
[gentoo-portage-dev] Re: [PATCH] ecompress: optimize docompress -x precompressed comparison
On 6/28/20 12:54 PM, Zac Medico wrote:
> + LC_COLLATE=C sort -zu "${T}/.ecompress_skip_files" >
> "${T}/.ecompress_skip_files_sorted"|| die
> + LC_COLLATE=C sort -zu
> "${T}/.ecompress_had_precompressed" >
> "${T}/.ecompress_had_precompressed_sorted" || die
> + LC_COLLATE=C comm -z13
> "${T}/.ecompress_skip_files_sorted"
> "${T}/.ecompress_had_precompressed_sorted" >
> "${T}/.ecompress_had_precompressed" || die
I've updated my branch to use \n separators, since posix comm does not
support the -z option.
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
