Re: [gentoo-portage-dev] emerge-webrsync patch
On Thu, Feb 02, 2006 at 08:16:14AM +0100, Johannes Fahrenkrug wrote: Brian, I just want to make sure this is still on your agenda :) InSVN, and in the tree... :) ~harring pgpcmhNYyu2R8.pgp Description: PGP signature
Re: [gentoo-portage-dev] emerge-webrsync patch
Brian Harring wrote: On Thu, Feb 02, 2006 at 08:16:14AM +0100, Johannes Fahrenkrug wrote: Brian, I just want to make sure this is still on your agenda :) InSVN, and in the tree... :) Great! Thanks :) I was just wondering: Will I be mentioned anywhere? - Johannes. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
On Thu, Feb 02, 2006 at 01:30:58PM +0100, Johannes Fahrenkrug wrote: Brian Harring wrote: On Thu, Feb 02, 2006 at 08:16:14AM +0100, Johannes Fahrenkrug wrote: Brian, I just want to make sure this is still on your agenda :) InSVN, and in the tree... :) Great! Thanks :) I was just wondering: Will I be mentioned anywhere? Bah, having it fixed doesn't count? see rev2624; the changes are only in trunk (2.1), won't be in 2.0.54 most likely since 2.0.54 shuld be bug fixes only... ~harring pgp7CaImWy9Bh.pgp Description: PGP signature
Re: [gentoo-portage-dev] emerge-webrsync patch
Brian Harring wrote: On Thu, Feb 02, 2006 at 01:30:58PM +0100, Johannes Fahrenkrug wrote: Brian Harring wrote: On Thu, Feb 02, 2006 at 08:16:14AM +0100, Johannes Fahrenkrug wrote: Brian, I just want to make sure this is still on your agenda :) InSVN, and in the tree... :) Great! Thanks :) I was just wondering: Will I be mentioned anywhere? Bah, having it fixed doesn't count? Sure it does. But having your name appear somewhere in the sources of the greatest Linux distro has a certain appeal to it ;-) see rev2624; the changes are only in trunk (2.1), won't be in 2.0.54 most likely since 2.0.54 shuld be bug fixes only... Alrighty. - Johannes. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
Brian, I just want to make sure this is still on your agenda :) - Johannes. Brian Harring wrote: On Mon, Jan 23, 2006 at 12:39:06PM +0100, Johannes Fahrenkrug wrote: Brian, did you have a chance to look at this yet? Will be integrating the changes sometime this week (spaced it tbh) :) ~harring -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
Brian, did you have a chance to look at this yet? - Johannes. Johannes Fahrenkrug wrote: Brian Harring wrote: On Wed, Jan 11, 2006 at 10:15:00AM +0100, Johannes Fahrenkrug wrote: if [[ -n $PORTAGE_NICENESS ]] ! [[ -z $WE_ARE_NICED ]]; then Haven't looked at the patch yet, but a bit of bash fu for ya- [[ -n $VAR ]] == ! [[ -z $VAR ]] -z is zero length or unset, -n is length = 1 (thus must be set). Exactly! This code ... If [[ -n $VAR ]] == ! [[ -z $VAR ]] (which it is) then export WE_ARE_NICED=1 exec nice -n $PORTAGE_NICENESS $0 $@ fi ...has the purpose to execute the current script again with nice if PORTAGE_NICENESS is set and WE_ARE_NICED is not set (to avoid an infinite loop). Right? If... if [[ -n $PORTAGE_NICENESS ]] ! [[ -z $WE_ARE_NICED ]]; then ...is the same as... if [[ -n $PORTAGE_NICENESS ]] [[ -n $WE_ARE_NICED ]]; then ...then that would mean that the condition is only met when both PORTAGE_NICENESS and WE_ARE_NICED are set. But WE_ARE_NICED is only set inside the if-body. So I think the condition should read if [[ -n $PORTAGE_NICENESS ]] [[ -z $WE_ARE_NICED ]]; then meaning the condition is only met when PORTAGE_NICENESS is set and WE_ARE_NICED is NOT set. Maybe I'm completely wrong here, but I think that's the only way the condition makes sense. Please enlighten me if I'm wrong :) - Johannes. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
Great! Thank you :) I'll be watching the GWN closely :) - Johannes. Brian Harring wrote: On Mon, Jan 23, 2006 at 12:39:06PM +0100, Johannes Fahrenkrug wrote: Brian, did you have a chance to look at this yet? Will be integrating the changes sometime this week (spaced it tbh) :) ~harring -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
Brian Harring wrote: On Wed, Jan 11, 2006 at 10:15:00AM +0100, Johannes Fahrenkrug wrote: if [[ -n $PORTAGE_NICENESS ]] ! [[ -z $WE_ARE_NICED ]]; then Haven't looked at the patch yet, but a bit of bash fu for ya- [[ -n $VAR ]] == ! [[ -z $VAR ]] -z is zero length or unset, -n is length = 1 (thus must be set). Exactly! This code ... If [[ -n $VAR ]] == ! [[ -z $VAR ]] (which it is) then export WE_ARE_NICED=1 exec nice -n $PORTAGE_NICENESS $0 $@ fi ...has the purpose to execute the current script again with nice if PORTAGE_NICENESS is set and WE_ARE_NICED is not set (to avoid an infinite loop). Right? If... if [[ -n $PORTAGE_NICENESS ]] ! [[ -z $WE_ARE_NICED ]]; then ...is the same as... if [[ -n $PORTAGE_NICENESS ]] [[ -n $WE_ARE_NICED ]]; then ...then that would mean that the condition is only met when both PORTAGE_NICENESS and WE_ARE_NICED are set. But WE_ARE_NICED is only set inside the if-body. So I think the condition should read if [[ -n $PORTAGE_NICENESS ]] [[ -z $WE_ARE_NICED ]]; then meaning the condition is only met when PORTAGE_NICENESS is set and WE_ARE_NICED is NOT set. Maybe I'm completely wrong here, but I think that's the only way the condition makes sense. Please enlighten me if I'm wrong :) - Johannes. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
On Wed, Jan 11, 2006 at 10:15:00AM +0100, Johannes Fahrenkrug wrote: if [[ -n $PORTAGE_NICENESS ]] ! [[ -z $WE_ARE_NICED ]]; then Haven't looked at the patch yet, but a bit of bash fu for ya- [[ -n $VAR ]] == ! [[ -z $VAR ]] -z is zero length or unset, -n is length = 1 (thus must be set). ~harring pgpYYLVtoPlnG.pgp Description: PGP signature
Re: [gentoo-portage-dev] emerge-webrsync patch
Ok, here's the patch for emerge-delta-webrync to use PORTAGE_NICENESS.
Brian, I hope this is what you had in mind.
I didn't alter the emerge-webrsync patch again, I think it's OK the way
I altered it in the last patch I posted,
but I attached it again anyway.
- Johannes.
--- /usr/bin/emerge-delta-webrsync 2006-01-04 09:26:07.0 +0100
+++ emerge-delta-webrsync 2006-01-04 09:25:43.068355800 +0100
@@ -7,8 +7,12 @@
#---
#initialization
#--
+PORTAGE_NICENESS=$(portageq envvar PORTAGE_NICENESS)
+if [ -z $PORTAGE_NICENESS ] ; then
+PORTAGE_NICENESS=0
+fi
-f=$(python -c'import portage; print |.join([portage.settings[x] for x in (GENTOO_MIRRORS, PORTDIR, FETCHCOMMAND, USERLAND, DISTDIR, PORTAGE_TMPDIR)])')
+f=$(nice -n $PORTAGE_NICENESS python -c'import portage; print |.join([portage.settings[x] for x in (GENTOO_MIRRORS, PORTDIR, FETCHCOMMAND, USERLAND, DISTDIR, PORTAGE_TMPDIR)])')
IFS='|'
GENTOO_MIRRORS=${f%%|*} ; f=${f#*|}
@@ -183,7 +187,7 @@
echo Syncing local tree...
if type -p tarsync /dev/null; then
echo apparently you have tarsync installed. using it.
- if ! tarsync ${FILE} ${PORTDIR} -v -s 1 -o portage -g portage -e /distfiles -e /packages -e /local; then
+ if ! nice -n $PORTAGE_NICENESS tarsync ${FILE} ${PORTDIR} -v -s 1 -o portage -g portage -e /distfiles -e /packages -e /local; then
echo ok, tarsync failed. that's teh suck :/
exit 6
fi
@@ -201,7 +205,7 @@
# Make sure user and group file ownership is root
chown -R 0:0 portage
cd portage
- rsync -av --progress --stats --delete --delete-after \
+ nice -n $PORTAGE_NICENESS rsync -av --progress --stats --delete --delete-after \
--exclude='/distfiles' --exclude='/packages' \
--exclude='/local' . ${PORTDIR%%/}
cd ..
--- /usr/sbin/emerge-webrsync 2005-12-28 11:49:18.0 +0100
+++ emerge-webrsync 2005-12-28 17:33:50.179723896 +0100
@@ -18,9 +18,19 @@
cd $DISTDIR
found=0
+rsyncops=-av --progress
+
+PORTAGE_NICENESS=$(portageq envvar PORTAGE_NICENESS)
+if [ -z $PORTAGE_NICENESS ] ; then
+PORTAGE_NICENESS=0
+fi
+
if [ $1 == -v ] ; then
wgetops=
else
+if [ $1 == -q ] ; then
+ rsyncops=-aq
+fi
#this sucks. probably better to do 1 /dev/null
#that said, waiting on the refactoring.
if [ ${FETCHCOMMAND/wget} != ${FETCHCOMMAND} ]; then
@@ -42,7 +52,7 @@
sync_local() {
echo Syncing local tree...
- if ! tar jxf $FILE; then
+ if ! nice -n $PORTAGE_NICENESS tar jxf $FILE; then
echo Tar failed to extract the image. Please review the output.
echo Executed command: tar jxf $FILE
exit 1
@@ -51,14 +61,14 @@
# Make sure user and group file ownership is root
chown -R 0:0 portage
cd portage
- rsync -av --progress --stats --delete --delete-after \
+ nice -n $PORTAGE_NICENESS rsync ${rsyncops} --stats --delete --delete-after \
--exclude='/distfiles' --exclude='/packages' \
--exclude='/local' . ${PORTDIR%%/}
cd ..
echo cleaning up
rm -rf portage
echo transferring metadata/cache
- emerge metadata
+ nice -n $PORTAGE_NICENESS emerge metadata
}
echo Fetching most recent snapshot
Re: [gentoo-portage-dev] emerge-webrsync patch
Brian Harring wrote: snip I'd also raid the tarsync call- this is something I was intending on doing but have't yet. /snip I don't have a very deep knowledge of all the portage internals. Are you suggesting that I should rather leave this up to you since you were planning on changing these things anyway? - Johannes. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
On Thu, Dec 29, 2005 at 03:51:03PM +0100, Johannes Fahrenkrug wrote: Brian Harring wrote: snip I'd also raid the tarsync call- this is something I was intending on doing but have't yet. /snip I don't have a very deep knowledge of all the portage internals. Are you suggesting that I should rather leave this up to you since you were planning on changing these things anyway? Nah, I'm suggesting you do it since I don't have the time right now :) Mods involved are pretty straightforward- pretty much just flip through emerge-delta-webrsync, look at the python -c call, and tarsync call. Integrating PORTAGE_NICENESS in, well, you already know how to do it. :) I'll get if others don't, but it'll be next week before I have time to take a stab at it. ~harring pgpPUncGUdFN5.pgp Description: PGP signature
Re: [gentoo-portage-dev] emerge-webrsync patch
On Wed, 2005-12-28 at 13:04 +0100, Johannes Fahrenkrug wrote: I put a nice -n 19 in front of the tar, rsync and emerge metadata commands because normally calling emerge-webrsync renders my box unusable for 15 to 20 minutes. You still notice a difference when using nice but everything seems to be at least noticably smoother than before. Instead of hardcoding the nice value, use PORTAGE_NICENESS. Here is how it is done in revdep-rebuild # Obey PORTAGE_NICENESS PORTAGE_NICENESS=$(portageq envvar PORTAGE_NICENESS) [ ! -z $PORTAGE_NICENESS ] renice $PORTAGE_NICENESS $$ /dev/null Regards, Paul -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] emerge-webrsync patch
Paul Varner wrote:
Instead of hardcoding the nice value, use PORTAGE_NICENESS. Here is how
it is done in revdep-rebuild
# Obey PORTAGE_NICENESS
PORTAGE_NICENESS=$(portageq envvar PORTAGE_NICENESS)
[ ! -z $PORTAGE_NICENESS ] renice $PORTAGE_NICENESS $$ /dev/null
Good point. Is this patch better? Or should it rather be _exactly_ as it
is in revdep-rebuild?
- Johannes.
--- /usr/sbin/emerge-webrsync 2005-12-28 11:49:18.0 +0100
+++ emerge-webrsync 2005-12-28 17:33:50.179723896 +0100
@@ -18,9 +18,19 @@
cd $DISTDIR
found=0
+rsyncops=-av --progress
+
+PORTAGE_NICENESS=$(portageq envvar PORTAGE_NICENESS)
+if [ -z $PORTAGE_NICENESS ] ; then
+PORTAGE_NICENESS=0
+fi
+
if [ $1 == -v ] ; then
wgetops=
else
+if [ $1 == -q ] ; then
+ rsyncops=-aq
+fi
#this sucks. probably better to do 1 /dev/null
#that said, waiting on the refactoring.
if [ ${FETCHCOMMAND/wget} != ${FETCHCOMMAND} ]; then
@@ -42,7 +52,7 @@
sync_local() {
echo Syncing local tree...
- if ! tar jxf $FILE; then
+ if ! nice -n $PORTAGE_NICENESS tar jxf $FILE; then
echo Tar failed to extract the image. Please review the output.
echo Executed command: tar jxf $FILE
exit 1
@@ -51,14 +61,14 @@
# Make sure user and group file ownership is root
chown -R 0:0 portage
cd portage
- rsync -av --progress --stats --delete --delete-after \
+ nice -n $PORTAGE_NICENESS rsync ${rsyncops} --stats --delete --delete-after \
--exclude='/distfiles' --exclude='/packages' \
--exclude='/local' . ${PORTDIR%%/}
cd ..
echo cleaning up
rm -rf portage
echo transferring metadata/cache
- emerge metadata
+ nice -n $PORTAGE_NICENESS emerge metadata
}
echo Fetching most recent snapshot
Re: [gentoo-portage-dev] emerge-webrsync patch
On Wed, Dec 28, 2005 at 05:38:02PM +0100, Johannes Fahrenkrug wrote: Paul Varner wrote: Instead of hardcoding the nice value, use PORTAGE_NICENESS. Here is how it is done in revdep-rebuild # Obey PORTAGE_NICENESS PORTAGE_NICENESS=$(portageq envvar PORTAGE_NICENESS) [ ! -z $PORTAGE_NICENESS ] renice $PORTAGE_NICENESS $$ /dev/null Good point. Is this patch better? Or should it rather be _exactly_ as it is in revdep-rebuild? I'd suggest raiding from emerge-delta-webrsync for the portageq call; it's a bit nasty, but it's a single call rather then multiple. I'd also raid the tarsync call- this is something I was intending on doing but have't yet. It will cut out the untarring/rsyncing call to 2 read throughs of the tarball, and single run through the tree. Fair bit faster, especially if the user's box doesn't have the ram to buffer the tree/tarball in memory. Tagging portage_niceness into it, just create a var with the appropriate nice call- if no PORTAGE_NICENESS, then the var is empty. ~harring pgpNUrhTiBo25.pgp Description: PGP signature
Re: [gentoo-portage-dev] emerge-webrsync patch
On Wed, 2005-12-28 at 17:38 +0100, Johannes Fahrenkrug wrote: Good point. Is this patch better? Or should it rather be _exactly_ as it is in revdep-rebuild? I personally would do it the same way as revdep-rebuild since that causes the entire script and anything it calls to be run at the value set in PORTAGE_NICENESS. However, the way you have done it works just as well. As far as your patch goes, you don't need to call the emerge metadata with a nice value since the emerge command reads and uses the PORTAGE_NICENESS value. Finally, I am not a portage developer, so if this gets accepted and used depends upon them and not me. Regards, Paul -- gentoo-portage-dev@gentoo.org mailing list
