Re: [gentoo-portage-dev] Re: running ebuild in src tree
On Thu, Mar 12, 2015 at 10:26 AM, Joakim Tjernlund
joakim.tjernl...@transmode.se wrote:
On Thu, 2015-03-12 at 01:27 +, Duncan wrote:
Zac Medico posted on Wed, 11 Mar 2015 12:03:10 -0700 as excerpted:
On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
When developing code it would be really nice if one could run
your ebuild on that src tree as is(no
fetch, unpack etc.)
The existing convention is to create an ebuild with version
and use one of the live vcs eclasses
such as git-r3 to pull the live sources in the src_unpack
function. In a future EAPI, we plan to add
some features related to this [1].
I think you misunderstand, [1] is not what I want to do(I think):
Got my src working copy and made a few modds, not commitet yet. Now
I just want build/test etc. before
committing and to do that I just run
mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my
code is built which I already have in mytree.
Well, you can create a - ebuild that copies your sources from
$directory to $WORKDIR. Maybe use an
environment to configure whether it pulls from a local directory or a
vcs repository.
@ Joakim T:
FWIW, a commonly recommended user-level portage optimization is to point
$PORTAGE_TMPDIR at a tmpfs mount. As long as you have sufficient memory,
that lets all building take place in the tmpfs and thus in memory,
eliminating many read-accesses and
most/all write accesses to permanent storage during the build and
(fake-)install phases.
In addition to speeding up emerge itself, this reduces build-time I/O,
which often becomes the bottleneck
on which other processes may be waiting as well, thus allowing other
processes more efficient access to
permanent storage while emerge is ongoing. Between this and setting
PORTAGE_NICENESS=20, emerge is /much/
better behaved during builds, interrupting other processes much less
and thus letting you carry on with
other things while emerge is doing its thing, with far less interruption.
=:^)
For instance, here I have /tmp as a tmpfs mount (with /var/tmp being a
bind-mount of the same tmpfs), and
in make.conf, have the line:
PORTAGE_TMPDIR=/tmp
Emerge then creates /tmp/portage, and within it, creates the usual cat/
pkg/ build trees, etc, as it
emerges various packages.
Obviously, your sources in permanent storage are going to be cache-copied
into memory as you do the build anyway, and pointing PORTAGE_TMPDIR at
tmpfs then becomes a copy to
(tmpfs) memory only. While that doesn't technically eliminate the
copies (since the read into tmpfs will
cache and you'll have the tmpfs copy as well), it DOES mean most of the
work beyond the initial read into
memory will be memory-only, so you DO eliminate the permanent-storage
copies.
Is that sufficiently close to what you were looking to do? Beyond that,
as Zac suggests, just have the ebuild grab the sources from wherever you
put them as your src_unpack, as at that point it'll be a copy to tmpfs,
and thus take essentially the same
time (or even less since it'll avoid the build-time writes to permanent
storage) as doing the in-place
build directly. Plus, creating a tmpfs mount if necessary, and
setting PORTAGE_TMPDIR, is easy, and
you'll dramatically speed-up normal builds as well. =:^)
No, there can be no copy of sources for what I want. It just gets in the
way having to do that.
Hacks like this seems to work:
I wouldn't really call it a hack either, but whichever ;)
post_src_compile() {
# make it compile every time
rm -f ${PORTAGE_BUILDDIR}/.compiled
}
post_src_install() {
# make it install every time
rm -f ${PORTAGE_BUILDDIR}/.installed
}
#hmm, doesn't seem to be an post_package function
#post_package() {
# rm -f ${PORTAGE_BUILDDIR}/.packaged
#}
src_unpack() {
#dir need to exist
mkdir -p ${S} || die
}
src_compile() {
EBUILDDIR=$(dirname ${FILESDIR})
MYTRUNK=${EBUILDDIR}/../../..
# add sandbox exceptions
addwrite ${MYTRUNK}
cd ${MYTRUNK} || die
cd ${PN}
.
}
I'm a bit curious what sort of workflow you are trying to achieve here is.
Which build artifacts are you looking for; the actual built binaries or the
built package that is merged into the livefs?
If the artifacts are packages, then I think this approach makes some
sense...but otherwise I'd just go into trunk and run the build process
(what does the ebuild gain you here?)
-A
Re: [gentoo-portage-dev] Re: running ebuild in src tree
On Fri, 2015-03-13 at 06:32 +, Duncan wrote: Joakim Tjernlund posted on Thu, 12 Mar 2015 17:26:59 + as excerpted: No, there can be no copy of sources for what I want. It just gets in the way having to do that. ?? Copying to tmpfs is copying to memory, and copying to memory in /some/ form must occur in ordered to operate on the files at all, that is, with any build at all, so I don't see the problem, at least if you're working on a machine with say 2+ gigs RAM. (There might be problems on a half-gig RAM machine, but that's not particularly appropriate as a developer machine in any case, these days, unless you're talking embedded, which you didn't mention.) But of course, gentoo/portage lets you do it your way too, as demonstrated by the hacks you posted. =:^) When you are developing SW you do the edit/build cycle often and it is really annoying to copy a big src tree, rebuild everything (as timestamps, deps changes), move to another src to do any debugging etc. Try it on your own development by just copying you src tree every time you want to build, you get very tired of it real soon :) Don't get me wrong, portage is a wounderful pkg mgr, I just want to extend its use to be more helpful during development too, without having to do really ugly hacks like I showed you. Jocke
Re: [gentoo-portage-dev] Re: running ebuild in src tree
On Thu, 2015-03-12 at 01:27 +, Duncan wrote:
Zac Medico posted on Wed, 11 Mar 2015 12:03:10 -0700 as excerpted:
On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
When developing code it would be really nice if one could run your
ebuild on that src tree as is(no
fetch, unpack etc.)
The existing convention is to create an ebuild with version and
use one of the live vcs eclasses
such as git-r3 to pull the live sources in the src_unpack function. In
a future EAPI, we plan to add
some features related to this [1].
I think you misunderstand, [1] is not what I want to do(I think):
Got my src working copy and made a few modds, not commitet yet. Now I
just want build/test etc. before
committing and to do that I just run
mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my
code is built which I already have in mytree.
Well, you can create a - ebuild that copies your sources from
$directory to $WORKDIR. Maybe use an
environment to configure whether it pulls from a local directory or a vcs
repository.
@ Joakim T:
FWIW, a commonly recommended user-level portage optimization is to point
$PORTAGE_TMPDIR at a tmpfs mount. As long as you have sufficient memory,
that lets all building take place in the tmpfs and thus in memory,
eliminating many read-accesses and
most/all write accesses to permanent storage during the build and
(fake-)install phases.
In addition to speeding up emerge itself, this reduces build-time I/O, which
often becomes the bottleneck
on which other processes may be waiting as well, thus allowing other
processes more efficient access to
permanent storage while emerge is ongoing. Between this and setting
PORTAGE_NICENESS=20, emerge is /much/
better behaved during builds, interrupting other processes much less and
thus letting you carry on with
other things while emerge is doing its thing, with far less interruption.
=:^)
For instance, here I have /tmp as a tmpfs mount (with /var/tmp being a
bind-mount of the same tmpfs), and
in make.conf, have the line:
PORTAGE_TMPDIR=/tmp
Emerge then creates /tmp/portage, and within it, creates the usual cat/ pkg/
build trees, etc, as it
emerges various packages.
Obviously, your sources in permanent storage are going to be cache-copied
into memory as you do the build anyway, and pointing PORTAGE_TMPDIR at tmpfs
then becomes a copy to
(tmpfs) memory only. While that doesn't technically eliminate the copies
(since the read into tmpfs will
cache and you'll have the tmpfs copy as well), it DOES mean most of the work
beyond the initial read into
memory will be memory-only, so you DO eliminate the permanent-storage copies.
Is that sufficiently close to what you were looking to do? Beyond that,
as Zac suggests, just have the ebuild grab the sources from wherever you
put them as your src_unpack, as at that point it'll be a copy to tmpfs, and
thus take essentially the same
time (or even less since it'll avoid the build-time writes to permanent
storage) as doing the in-place
build directly. Plus, creating a tmpfs mount if necessary, and setting
PORTAGE_TMPDIR, is easy, and
you'll dramatically speed-up normal builds as well. =:^)
No, there can be no copy of sources for what I want. It just gets in the way
having to do that.
Hacks like this seems to work:
post_src_compile() {
# make it compile every time
rm -f ${PORTAGE_BUILDDIR}/.compiled
}
post_src_install() {
# make it install every time
rm -f ${PORTAGE_BUILDDIR}/.installed
}
#hmm, doesn't seem to be an post_package function
#post_package() {
# rm -f ${PORTAGE_BUILDDIR}/.packaged
#}
src_unpack() {
#dir need to exist
mkdir -p ${S} || die
}
src_compile() {
EBUILDDIR=$(dirname ${FILESDIR})
MYTRUNK=${EBUILDDIR}/../../..
# add sandbox exceptions
addwrite ${MYTRUNK}
cd ${MYTRUNK} || die
cd ${PN}
.
}
