Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 09/06/2012 02:50 PM, Michał Górny wrote:
On Thu, 6 Sep 2012 09:49:13 -0700
Brian Harring ferri...@gmail.com wrote:
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Realizing this is a late response I would like to add Um, what?
src_fetch should only be putting things into /usr/portage/distfiles,
never into ${WORKDIR}, that's for src_unpack to handle.
Am I missing something here?
- -Zero
Basically that restriction would be interpretted as $WORKDIR must be
setup/preserved from invocation of src_fetch to actual building.
Via that restrict, both scenarios should be addressed in full.
Does separate src_fetch() provide any benefit in that scenario?
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJQT7z+AAoJEKXdFCfdEflKVPQQALGcLGAgfo8U+M6TdW5Edksf
oqaXE+NSTeFe2DE0G2mgKYSdSIZgMiFp5mLFwpdfAT1gjzFAc+34+5SY8X/0uaaG
OU7fafdUmOlqgD7rCvX56kSWZVPmTV3oZDghmwB1SUIQpL9PNSZoz5uKoatt4UL8
mBMiTmnsYou8f+wDCJoN8eLVoQb/Hm2inobGUCozCsqU6ASgk1eVePpAmJNNVKp6
wrKuVbj4FeDS17Q6xc2g8exXlkxhGmdS1MmugKBR9csYC9P82fh4bXqVzxG15h9O
YZDU5nagJQ9fY6M7oeKg6etVe6PvwOd/FH0Z4wtQJ2NicOsu6DiBf7J+yLvadJdF
M5fE1kxjtR+rm+6bfNgBl2hP5DPdUYxPnZChftPNRpiVe0P0YZGuRgy+GqfXSmMh
8Zf37hJylauBDy397yGapIJ4ergpYVb3Z1ZfU6uW7n8k0apqjqk+TYLEv2tajtTc
WBdRoBAT1LxvjFIHj2Bf5uzNtqev4l19vJv1AnALgs1v1Z8/TiSPB/B/2DvvIawR
Ys+mAEKzQOCezaCjVOpjq7pvi80/8PMcw6Txk9WpsAezrxdAj2X24kwsptVugCAO
lG3agqmOQgH+vCf0PkplxMSGBlofh4hZ3mNbuDaFlaRfiSlzFX//rkQTtOjK4YQu
6SiRnKxiDzJEn+Q1SNUr
=xgcT
-END PGP SIGNATURE-
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Tue, Sep 11, 2012 at 06:36:46PM -0400, Rick Zero_Chaos Farina wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 09/06/2012 02:50 PM, Micha?? G??rny wrote:
On Thu, 6 Sep 2012 09:49:13 -0700
Brian Harring ferri...@gmail.com wrote:
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Realizing this is a late response I would like to add Um, what?
src_fetch should only be putting things into /usr/portage/distfiles,
never into ${WORKDIR}, that's for src_unpack to handle.
Am I missing something here?
WORKDIR was an example; punting it directly into the pkgs distfiles is
also fine.
Basically, you're assuming that the content *is* cachable- cause if it
isn't, then dumping in ${DISTDIR} is wasteful (both since it'll
require a copy into the ebuild's workdir, and since it means
increasing crap accumulating in the workdir).
Further, there are cases where content *is* available, but is
fundamentally outside ${DISTDIR}- in cros, they have the
chrome/chromium source available in certain cases.
Now, either we can store 13GB int ${DISTDIR} that's a copy of that
external developer checkout, then copy that into ${WORKDIR}, or w/ a
restrict marker, copy that sucker into ${WORKDIR} directly.
Is it a corner case? Yep, but it's easy enough to deal w/- the only
constraint there is that the src_fetch's target dir isn't ${DISTDIR},
it's ${WORKDIR}, and the PM is required to preserve ${WORKDIR} from
src_fetch to the time of that pkg actually running.
~harring
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
CM == Ciaran McCreesh ciaran.mccre...@googlemail.com writes: CM This doesn't work if we have, for example, foo:1 and foo:2 both using CM the same SCM repository, but different branches. The subversion eclass already handles that; it stores in $distfiles/$P/$branch. The cvs eclass also could do so. -JimC -- James Cloos cl...@jhcloos.com OpenPGP: 1024D/ED7DAEA6
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, Sep 05, 2012 at 12:07:22PM +0100, Ciaran McCreesh wrote:
On Wed, 5 Sep 2012 13:00:05 +0200
Micha?? G??rny mgo...@gentoo.org wrote:
I guess that's a pretty comprehensive we need to do this properly
then.
Did I say we don't need to? We have the two eclasses which need to do
this properly, right? So what's your problem?
The problem is that we need to work out how to deal with this at least
for Subversion, and probably for CVS too... As much as we'd like to, we
can't just roll out a src_fetch without eclass changes. This doesn't
appear to be a trivial feature to provide.
You're conflating the required phase/hook w/ existing bad
implementations; they're separate.
The bad implementations can't use the hook till they sort out their
mess; straight forward enough.
Worth noting, last I looked, git eclass actually didn't do this right
either; basically merges the namespace of each remote/fetch source
into the local namespace, no prefixing. Fixable, but the issue mostly
comes about because of the fact we do *not* have a src_fetch in the
first place.
Either way, if the hook was in place, I expect the eclass level issues
to be sorted shortly after; right now there isn't a reason to do that
work (chicken/egg).
Consider that my +1 for src_fetch also, although FDEPEND is needed or
dependencies; I don't care which, getting a src_fetch has been on my
todo for chrome-os for a while now.
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Basically that restriction would be interpretted as $WORKDIR must be
setup/preserved from invocation of src_fetch to actual building.
Via that restrict, both scenarios should be addressed in full.
~harring
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Thu, 6 Sep 2012 09:49:13 -0700
Brian Harring ferri...@gmail.com wrote:
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Basically that restriction would be interpretted as $WORKDIR must be
setup/preserved from invocation of src_fetch to actual building.
Via that restrict, both scenarios should be addressed in full.
Does separate src_fetch() provide any benefit in that scenario?
--
Best regards,
Michał Górny
signature.asc
Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
Yes. The manager can still parallelize prefetching, only consuming a build
job slot post fetch.
On Sep 6, 2012 11:49 AM, Michał Górny mgo...@gentoo.org wrote:
On Thu, 6 Sep 2012 09:49:13 -0700
Brian Harring ferri...@gmail.com wrote:
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Basically that restriction would be interpretted as $WORKDIR must be
setup/preserved from invocation of src_fetch to actual building.
Via that restrict, both scenarios should be addressed in full.
Does separate src_fetch() provide any benefit in that scenario?
--
Best regards,
Michał Górny
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Tue, 4 Sep 2012 19:23:51 +0200 Michał Górny mgo...@gentoo.org wrote: The 'checking out' language for src_unpack() sounds like it assumes a DVCS such as mercurial or git. What about cvs or svn, where fetching is also checking out? (This is probably a trivial thing to clear up, though.) They either stay with src_unpack() or do 'cvs up' in src_fetch() and just copy files over in src_unpack(). Anyway, that's what they do now -- update the copy in distfiles/cvs-src and then copy it. This doesn't work if we have, for example, foo:1 and foo:2 both using the same SCM repository, but different branches. Much as we'd like to pretend that everyone uses Git, we can't really ignore this case... So we have to decide: do we make the src_fetch copy the data somewhere after all, or do we require that eclasses do something obscene to avoid this? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 5 Sep 2012 08:25:54 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 4 Sep 2012 19:23:51 +0200 Michał Górny mgo...@gentoo.org wrote: The 'checking out' language for src_unpack() sounds like it assumes a DVCS such as mercurial or git. What about cvs or svn, where fetching is also checking out? (This is probably a trivial thing to clear up, though.) They either stay with src_unpack() or do 'cvs up' in src_fetch() and just copy files over in src_unpack(). Anyway, that's what they do now -- update the copy in distfiles/cvs-src and then copy it. This doesn't work if we have, for example, foo:1 and foo:2 both using the same SCM repository, but different branches. Much as we'd like to pretend that everyone uses Git, we can't really ignore this case... So we have to decide: do we make the src_fetch copy the data somewhere after all, or do we require that eclasses do something obscene to avoid this? Eclasses have to handle themselves, if we are supporting this. There's no point in bloating the solution for 1 theoretical package on 1000 which doesn't even exist. And yes, it is *very* unlikely that someone uses a slotted live ebuild with two branches being meaningful and managed in the same repo. Even if such thing exists, it is broken anyway because you can't say that re-fetching the branches back and forth is a correct solution. And it breaks existing tools anyway. Well, even I doubt eclasses need to do something obscene. The need for that is so small that I believe if someone was crazy enough, he could just set an appropriate storedir in ebuild. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 5 Sep 2012, Michał Górny wrote: And yes, it is *very* unlikely that someone uses a slotted live ebuild with two branches being meaningful and managed in the same repo. We do that all the time for app-editors/emacs-vcs. For example, we used to have live ebuilds for the trunk and for the emacs-24 branch, sharing a common repository. Even if such thing exists, it is broken anyway because you can't say that re-fetching the branches back and forth is a correct solution. And it breaks existing tools anyway. I haven't notice any breakage with it. Ulrich
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 5 Sep 2012 11:07:44 +0200 Ulrich Mueller u...@gentoo.org wrote: On Wed, 5 Sep 2012, Michał Górny wrote: And yes, it is *very* unlikely that someone uses a slotted live ebuild with two branches being meaningful and managed in the same repo. We do that all the time for app-editors/emacs-vcs. For example, we used to have live ebuilds for the trunk and for the emacs-24 branch, sharing a common repository. Doesn't bzr actually work alike git? i.e. keep all commits when switching branches? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 5 Sep 2012, Michał Górny wrote: On Wed, 5 Sep 2012 11:07:44 +0200 Ulrich Mueller u...@gentoo.org wrote: On Wed, 5 Sep 2012, Michał Górny wrote: And yes, it is *very* unlikely that someone uses a slotted live ebuild with two branches being meaningful and managed in the same repo. We do that all the time for app-editors/emacs-vcs. For example, we used to have live ebuilds for the trunk and for the emacs-24 branch, sharing a common repository. Doesn't bzr actually work alike git? i.e. keep all commits when switching branches? Actually, bzr.eclass uses branches without a working tree (what would be called a bare repository in the Git world). There's no need to switch branches. Ulrich
Re: Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
And yes, it is *very* unlikely that someone uses a slotted live ebuild with two branches being meaningful and managed in the same repo. Even if such thing exists, it is broken anyway because you can't say that re-fetching the branches back and forth is a correct solution. And it breaks existing tools anyway. This is done large-scale for all KDE ebuilds (in the KDE overlay) to support master and KDE/4.x stable branch. Most use git, so no problem; some (still) use subversion but will be migrated upstream soon(?). Other examples are libreoffice (main tree, git) and cups (main tree, subversion). -- Andreas K. Huettel Gentoo Linux developer kde, sci, arm, tex, printing signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 05 Sep 2012 12:45:16 +0200 Andreas K. Huettel dilfri...@gentoo.org wrote: And yes, it is *very* unlikely that someone uses a slotted live ebuild with two branches being meaningful and managed in the same repo. Even if such thing exists, it is broken anyway because you can't say that re-fetching the branches back and forth is a correct solution. And it breaks existing tools anyway. This is done large-scale for all KDE ebuilds (in the KDE overlay) to support master and KDE/4.x stable branch. Most use git, so no problem; some (still) use subversion but will be migrated upstream soon(?). Other examples are libreoffice (main tree, git) and cups (main tree, subversion). I guess that's a pretty comprehensive we need to do this properly then. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 5 Sep 2012 11:49:03 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Wed, 05 Sep 2012 12:45:16 +0200 Andreas K. Huettel dilfri...@gentoo.org wrote: And yes, it is *very* unlikely that someone uses a slotted live ebuild with two branches being meaningful and managed in the same repo. Even if such thing exists, it is broken anyway because you can't say that re-fetching the branches back and forth is a correct solution. And it breaks existing tools anyway. This is done large-scale for all KDE ebuilds (in the KDE overlay) to support master and KDE/4.x stable branch. Most use git, so no problem; some (still) use subversion but will be migrated upstream soon(?). Other examples are libreoffice (main tree, git) and cups (main tree, subversion). I guess that's a pretty comprehensive we need to do this properly then. Did I say we don't need to? We have the two eclasses which need to do this properly, right? So what's your problem? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 5 Sep 2012 13:00:05 +0200 Michał Górny mgo...@gentoo.org wrote: I guess that's a pretty comprehensive we need to do this properly then. Did I say we don't need to? We have the two eclasses which need to do this properly, right? So what's your problem? The problem is that we need to work out how to deal with this at least for Subversion, and probably for CVS too... As much as we'd like to, we can't just roll out a src_fetch without eclass changes. This doesn't appear to be a trivial feature to provide. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, 5 Sep 2012 12:07:22 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Wed, 5 Sep 2012 13:00:05 +0200 Michał Górny mgo...@gentoo.org wrote: I guess that's a pretty comprehensive we need to do this properly then. Did I say we don't need to? We have the two eclasses which need to do this properly, right? So what's your problem? The problem is that we need to work out how to deal with this at least for Subversion, and probably for CVS too... As much as we'd like to, we can't just roll out a src_fetch without eclass changes. This doesn't appear to be a trivial feature to provide. First of all, subversion isn't a problem here. Subversion doesn't have native branches. It's just directories in the tree, with tree having linear history. In other words, it just works. So it remains CVS and possibly darcs. But considering that the issue is so unlikely and so small, we can as well assume that the affected ebuilds should set ECVS_LOCALNAME to something with the branch. This is in line with how we handle SRC_URI collisions. And in any case, we may just decide to use a ECVS_LOCALNAME with branch appended by default. Finally, I don't think eclasses are really forced to use src_fetch() from day one. src_unpack() will still work for them, and we can adjust them gradually. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 05/09/12 07:25 AM, Michał Górny wrote: Finally, I don't think eclasses are really forced to use src_fetch() from day one. src_unpack() will still work for them, and we can adjust them gradually. ...except for the fact that the whole point of this is so that live ebuilds will download their sources via src_fetch() when emerge -f is called... :) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBHUtcACgkQ2ugaI38ACPA7YAEAmuGb+BY67admkDhAAO5sLDvI iChvTWdTFgRclPxeylEA/iUWFv3eCTzVBBhbvpGc44F2P8IO010OhEu1PrFj2mfC =fO5h -END PGP SIGNATURE-
[gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
Hello, As Sid Hayn raised today on #gentoo-portage, it would be useful to finally have portage able to fetch updates from VCS-es independently of src_unpack(). This could be used, for example, on machines temporarily connected to the network -- one would then fetch files while connected to the network, and perform the updates later. There are a few ways how we could handle that but the cleanest and most universal one seems to be defining a src_fetch() phase function in a future EAPI. In the EAPIs supporting src_fetch(), that phase function would be used by PM when requesting the files to be fetched. A default_src_fetch() will be declared as well, providing implementation-defined code fetching files like they are fetched now. Older EAPIs will simply always use that default. The phase function would be disjoint from the normal merge process, much like pkg_pretend(). In portage, it will be called as 'portage' user if FEATURES=userfetch is enabled. VCS eclasses supporting separated fetching would define two phase functions: - src_fetch() which would be responsible for fetching updates, - src_unpack() which would be responsible for checking out the source to work directory. The remaining issue is handling dependencies on the tools necessary to do fetching. For default_src_fetch(), we can assume that the package manager provides the necessary tools. For custom src_fetch(), we would need either to: 1) require satisfying whole DEPEND when fetching -- probably pointless, as it will make --fetchonly almost impossible when doing initial installs; 2) introduce a new dependency type (please do not get into details how we do it -- we will discuss that another time, at the moment please just keep it as 'new dependency type') -- and we probably end up having a switch for --fetchonly without installing deps (thus omitting packages where they are not satisfied), and with deps; 3) [ugly!] assume that src_fetch() should check for its deps and fail if they are not satisfied. If that's mostly for live ebuilds, it may be acceptable. Then the package manager will just have one 'fetch failed' on --fetchonly (or early pre-fetch), and it will have to invoke src_fetch() after satisfying the deps, before src_unpack(). What do you think? What are your ideas, suggestions? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/04/2012 12:43 PM, Michał Górny wrote: Hello, As Sid Hayn raised today on #gentoo-portage, it would be useful to If you insist on using real names mine is Rick ;-) finally have portage able to fetch updates from VCS-es independently of src_unpack(). This could be used, for example, on machines temporarily connected to the network -- one would then fetch files while connected to the network, and perform the updates later. There are a few ways how we could handle that but the cleanest and most universal one seems to be defining a src_fetch() phase function in a future EAPI. In the EAPIs supporting src_fetch(), that phase function would be used by PM when requesting the files to be fetched. A default_src_fetch() will be declared as well, providing implementation-defined code fetching files like they are fetched now. Older EAPIs will simply always use that default. The phase function would be disjoint from the normal merge process, much like pkg_pretend(). In portage, it will be called as 'portage' user if FEATURES=userfetch is enabled. VCS eclasses supporting separated fetching would define two phase functions: - src_fetch() which would be responsible for fetching updates, - src_unpack() which would be responsible for checking out the source to work directory. The remaining issue is handling dependencies on the tools necessary to do fetching. For default_src_fetch(), we can assume that the package manager provides the necessary tools. For custom src_fetch(), we would need either to: 1) require satisfying whole DEPEND when fetching -- probably pointless, as it will make --fetchonly almost impossible when doing initial installs; 2) introduce a new dependency type (please do not get into details how we do it -- we will discuss that another time, at the moment please just keep it as 'new dependency type') -- and we probably end up having a switch for --fetchonly without installing deps (thus omitting packages where they are not satisfied), and with deps; 3) [ugly!] assume that src_fetch() should check for its deps and fail if they are not satisfied. If that's mostly for live ebuilds, it may be acceptable. Then the package manager will just have one 'fetch failed' on --fetchonly (or early pre-fetch), and it will have to invoke src_fetch() after satisfying the deps, before src_unpack(). What do you think? What are your ideas, suggestions? -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJQRjQeAAoJEKXdFCfdEflKdDAP/jyV8+NwEBX+etKq7e+qzbVg oIgjqOLfqYEmbEkO9WMsUltsf8YATy9zGB1itRiLUb5v+eptjEDvp2dx/AA4YVio RLYwtMBJsESXd6d+3PbCTeoZHTZwb7ue6yY+qC0auyC4mD6nrkDY8swungcDwFHN GZ99gAtRnQ5h8+phrKHyWzePTkBN5+32GCIn2XfRwMo+T0tGyTeMGqYfPJPlEu0y LNJmdBU0TfzE7N8QA4sp/IDQQvBmZNpH6aQM+zkJWpvBUXBATdIPZZHBAy55TWaJ t9KxiHD6XF/h0ShlTIpsbfSj31FTil2l/LfhAQdXSrB4GPxLaoovOJb2vl9L+ZyP QJclNeL1kNaOO1MIm20tGJOV6Wh0iHsn/fJlbh4YYn4PaNF+F+UXyp69uGcniuCe PxFC/aosq39LoFRrZdyRfx3DWPXnhYfcFWRwEDosa/Qb/Mbljs+BZRRPIB2Y8ItV 9j5WokM6BkNPNeoNnaS1JvCaac7xiurizs7IaXxfYC5q8aZja1OdDrV9Jh7KxIUe q8rsKMk6y9KqGvSBfW3Y9JDgIdUUG8x0bVM/j9+gqOtPH5uFgPRnZxX/Hfb0+1J5 iLjFXgwLl2AtEuflY07KfKB9xyEV58x8gkCo/BUX8Y8E6re5/4o+kNiArQL2Z+YS SOD2BmvyVHOlurug9Lq4 =FOao -END PGP SIGNATURE-
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny mgo...@gentoo.org wrote: Hello, As Sid Hayn raised today on #gentoo-portage, it would be useful to finally have portage able to fetch updates from VCS-es independently of src_unpack(). This could be used, for example, on machines temporarily connected to the network -- one would then fetch files while connected to the network, and perform the updates later. There are a few ways how we could handle that but the cleanest and most universal one seems to be defining a src_fetch() phase function in a future EAPI. In the EAPIs supporting src_fetch(), that phase function would be used by PM when requesting the files to be fetched. A default_src_fetch() will be declared as well, providing implementation-defined code fetching files like they are fetched now. Older EAPIs will simply always use that default. The phase function would be disjoint from the normal merge process, much like pkg_pretend(). In portage, it will be called as 'portage' user if FEATURES=userfetch is enabled. VCS eclasses supporting separated fetching would define two phase functions: - src_fetch() which would be responsible for fetching updates, - src_unpack() which would be responsible for checking out the source to work directory. The 'checking out' language for src_unpack() sounds like it assumes a DVCS such as mercurial or git. What about cvs or svn, where fetching is also checking out? (This is probably a trivial thing to clear up, though.) Also, where would the local copy go? distfiles? It's common for distfiles to be stored on, e.g. an NFS mount, so you may need to be careful not to place repositories there which have filesystem semantics that are disagreeable to NFS. (The only example I know of off the top of my head is svn, where the documentation warns against using the dbd backend on top of NFS.) Other common remote mounts (such as cifs) may have restrictions that could force munging of filenames, too, and VCS infrastructures (or even unpacked checkouts with strange filenames) placed on those filesystems may have unanticipated results. It may be helpful to have some kind of adapter mount in place, or even generate a tarball of the local copy and store that instead. (That'd be problematic if multiple boxes were modifying the local copy on the same share, but that's obviously problematic anyway.) -- :wq
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/04/2012 12:43 PM, Michał Górny wrote: Hello, As Sid Hayn raised today on #gentoo-portage, it would be useful to finally have portage able to fetch updates from VCS-es independently of src_unpack(). This could be used, for example, on machines temporarily connected to the network -- one would then fetch files while connected to the network, and perform the updates later. There are a few ways how we could handle that but the cleanest and most universal one seems to be defining a src_fetch() phase function in a future EAPI. In the EAPIs supporting src_fetch(), that phase function would be used by PM when requesting the files to be fetched. A default_src_fetch() will be declared as well, providing implementation-defined code fetching files like they are fetched now. Older EAPIs will simply always use that default. The phase function would be disjoint from the normal merge process, much like pkg_pretend(). In portage, it will be called as 'portage' user if FEATURES=userfetch is enabled. VCS eclasses supporting separated fetching would define two phase functions: - src_fetch() which would be responsible for fetching updates, - src_unpack() which would be responsible for checking out the source to work directory. The remaining issue is handling dependencies on the tools necessary to do fetching. For default_src_fetch(), we can assume that the package manager provides the necessary tools. For custom src_fetch(), we would need either to: 1) require satisfying whole DEPEND when fetching -- probably pointless, as it will make --fetchonly almost impossible when doing initial installs; 2) introduce a new dependency type (please do not get into details how we do it -- we will discuss that another time, at the moment please just keep it as 'new dependency type') -- and we probably end up having a switch for --fetchonly without installing deps (thus omitting packages where they are not satisfied), and with deps; 3) [ugly!] assume that src_fetch() should check for its deps and fail if they are not satisfied. If that's mostly for live ebuilds, it may be acceptable. Then the package manager will just have one 'fetch failed' on --fetchonly (or early pre-fetch), and it will have to invoke src_fetch() after satisfying the deps, before src_unpack(). I believe the easiest (and honestly most sane) method is to simply have src_fetch in the live classes check for needed deps and die (with a please emerge blah) if deps are not found. Adding something like FDEPEND just seems to be getting way too crazy on the dependency tree AND would require things to build during fetch-only which doesn't make sense. Thanks, Zero What do you think? What are your ideas, suggestions? -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJQRjTjAAoJEKXdFCfdEflKGPoP+wYhzeFNcXO9Lv4qm5/oPXuC RIrCs2q6erWxowFbiTO//XILVbiH1nHBWx/uJV9S0hTM+dBRIh/zZaCXy5PYnLrK dTKiHgmpNPgTQ41QGI7BZl4lQFGLgGfdJnCncSTLmLZQtMGwXD4jZ8SQ/QE1wbwK rpHtYQ/Z5kUKFJG25MRFtFZ2ifZQVpVIPmmrFfyDh+1l9Oh8AtF6XKZTriwX1ppG osPz8jo2XtoYejqvD0kFEZwc5C7FKmULYyrB+tcl5dmOgf60LvcDTBkUGi7U2ewk DWhVoLX0zXWMZfoEi3c6GRsJgZ81yXIMfPC+SUYipdfBvdH6iWYuxfgJpSioUtbV s4xC4IyOGQehnt8OAe5PpHHJcqxlDVXidmVz0sKkmJB3dM6rIsamiNZajmKVrnyh 1zle8g4XBP8AFg/fB+OiOSuYNoI5GnC2D/rp7zAtajl+0GtVWvApPUFEa0DP8LlR bYXMIGpx+LAi9rYlEdNs3tW1C4OVjZpCKYU7cjNO15f2ZOVX4WNmAWtRmpoG6O5l zL1Mv6DRgdtoITfYfogupZBgjynHsCzKL1Mb2skqtmJ853ZjV4HGgXqxcica12O7 EoJ65EU5rrzCGSUz5rqunNwK9A11VBjxXWJ/e29NKt2vACxRuiwMXkl7EK2a/oCA pTBk5q6CvQFJCyzw2bee =ueCf -END PGP SIGNATURE-
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 09/04/2012 01:02 PM, Michael Mol wrote:
On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny mgo...@gentoo.org wrote:
Hello,
As Sid Hayn raised today on #gentoo-portage, it would be useful to
finally have portage able to fetch updates from VCS-es independently
of src_unpack(). This could be used, for example, on machines
temporarily connected to the network -- one would then fetch files
while connected to the network, and perform the updates later.
There are a few ways how we could handle that but the cleanest and most
universal one seems to be defining a src_fetch() phase function
in a future EAPI.
In the EAPIs supporting src_fetch(), that phase function would be used
by PM when requesting the files to be fetched. A default_src_fetch()
will be declared as well, providing implementation-defined code
fetching files like they are fetched now. Older EAPIs will simply
always use that default.
The phase function would be disjoint from the normal merge process,
much like pkg_pretend(). In portage, it will be called as 'portage'
user if FEATURES=userfetch is enabled.
VCS eclasses supporting separated fetching would define two phase
functions:
- src_fetch() which would be responsible for fetching updates,
- src_unpack() which would be responsible for checking out the source
to work directory.
The 'checking out' language for src_unpack() sounds like it assumes a
DVCS such as mercurial or git. What about cvs or svn, where fetching
is also checking out? (This is probably a trivial thing to clear up,
though.)
Also, where would the local copy go? distfiles? It's common for
distfiles to be stored on, e.g. an NFS mount, so you may need to be
careful not to place repositories there which have filesystem
semantics that are disagreeable to NFS. (The only example I know of
off the top of my head is svn, where the documentation warns against
using the dbd backend on top of NFS.)
Other common remote mounts (such as cifs) may have restrictions that
could force munging of filenames, too, and VCS infrastructures (or
even unpacked checkouts with strange filenames) placed on those
filesystems may have unanticipated results.
It may be helpful to have some kind of adapter mount in place, or even
generate a tarball of the local copy and store that instead. (That'd
be problematic if multiple boxes were modifying the local copy on the
same share, but that's obviously problematic anyway.)
All the live eclasses already drop a checkout (or whatever term you
like) of the repo in /usr/portage/distfiles. What we are talking about
here is separating the download/checkout into /usr/portage/distfiles
from the copy from /usr/portage/distfiles to ${S}. Making this two
separate phases would allow a reasonably sane emerge -f blah support
to fetch the live sources before build.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJQRjk9AAoJEKXdFCfdEflKeocQAJgSMaycMkvmdalYWUBU4lRJ
eNilSxWoMAuln+1FMRv9HxavQCMzkw/h9o3Lwy75Kv2+Z/MZy60caG7yplexSKEf
Z3apTwU1Cbj7WasDNkNRXgoHHsn55Fnr28lueFmcAlvMWDnBV0TmXCYFiRh961hN
7Co3wbHT4ahk1TTTURFDoWbE2R24e1UJFzrWRRO4oyoIDqPqOIOqhR+4Nnl5vsXa
/UQmvIqgzxlWgdjeaWceQxgexPt0g+ZwFUFIF6FhL3xqoF6kokBj64VxPRAgK05X
x78LSsul+A0H6sAU4S9e5CmZekxFyNanF3PJ5aZaAfp8dPTWaUFIhJHh3jDFktdT
zaO1LVpEOVnUenuV6XgpETN8XxThZHzXFRokuyR3Aza41/p3eg/jo07DGmUBLRP2
fu9h/ZVwrcPixgpTNd2+8YhxIxuoka7CmJiVb4ncAJ/Yp95G64dM57DXf46rKPTC
6LX/5jQgXcR+WbQswLumZJ+EJ7aIzGhJRFGXZmbNvWVrzFxfB7vtgUl6NUsKmzTo
gffrxrZexzlnvkCThrdM1C89BdAdkWATICiAAEkf2P9gzEdtB+p65VBqYcNzFc+Q
X9kw+X6lnAn6D/KqJdSAG9plhjRuZ9JTGnZ/ZlucyP7uVnmHV5+NOVn5MYvZ/Sso
z2F4QPIJi3C9LQkOSyKF
=vy05
-END PGP SIGNATURE-
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Tue, 4 Sep 2012 13:02:36 -0400 Michael Mol mike...@gmail.com wrote: On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny mgo...@gentoo.org wrote: Hello, As Sid Hayn raised today on #gentoo-portage, it would be useful to finally have portage able to fetch updates from VCS-es independently of src_unpack(). This could be used, for example, on machines temporarily connected to the network -- one would then fetch files while connected to the network, and perform the updates later. There are a few ways how we could handle that but the cleanest and most universal one seems to be defining a src_fetch() phase function in a future EAPI. In the EAPIs supporting src_fetch(), that phase function would be used by PM when requesting the files to be fetched. A default_src_fetch() will be declared as well, providing implementation-defined code fetching files like they are fetched now. Older EAPIs will simply always use that default. The phase function would be disjoint from the normal merge process, much like pkg_pretend(). In portage, it will be called as 'portage' user if FEATURES=userfetch is enabled. VCS eclasses supporting separated fetching would define two phase functions: - src_fetch() which would be responsible for fetching updates, - src_unpack() which would be responsible for checking out the source to work directory. The 'checking out' language for src_unpack() sounds like it assumes a DVCS such as mercurial or git. What about cvs or svn, where fetching is also checking out? (This is probably a trivial thing to clear up, though.) They either stay with src_unpack() or do 'cvs up' in src_fetch() and just copy files over in src_unpack(). Anyway, that's what they do now -- update the copy in distfiles/cvs-src and then copy it. Also, where would the local copy go? distfiles? It's common for distfiles to be stored on, e.g. an NFS mount, so you may need to be careful not to place repositories there which have filesystem semantics that are disagreeable to NFS. (The only example I know of off the top of my head is svn, where the documentation warns against using the dbd backend on top of NFS.) The actual code will be eclass-dependent, so it will go whatever it goes now. No magic, no exherbo. Just plain phase function. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On 09/04/2012 10:05 AM, Rick Zero_Chaos Farina wrote: I believe the easiest (and honestly most sane) method is to simply have src_fetch in the live classes check for needed deps and die (with a please emerge blah) if deps are not found. Adding something like FDEPEND just seems to be getting way too crazy on the dependency tree AND would require things to build during fetch-only which doesn't make sense. I think it's nicer to have FDEPEND because it makes the deps more complete, so the package manager can bail out when necessary, without even executing src_fetch. In the case of --fetchonly the package manager could simply bail out if the deps are not installed (like how it bails out for --buildpkgonly when the deps aren't installed). -- Thanks, Zac
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 04/09/12 01:32 PM, Zac Medico wrote: On 09/04/2012 10:05 AM, Rick Zero_Chaos Farina wrote: I believe the easiest (and honestly most sane) method is to simply have src_fetch in the live classes check for needed deps and die (with a please emerge blah) if deps are not found. Adding something like FDEPEND just seems to be getting way too crazy on the dependency tree AND would require things to build during fetch-only which doesn't make sense. I think it's nicer to have FDEPEND because it makes the deps more complete, so the package manager can bail out when necessary, without even executing src_fetch. In the case of --fetchonly the package manager could simply bail out if the deps are not installed (like how it bails out for --buildpkgonly when the deps aren't installed). Just looking into the future here; would things like archivers or other helpers used by src_unpack move to FDEPEND as well? or would this be limited solely to tools that data transfer? -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBGXQYACgkQ2ugaI38ACPDKGwEAlvqkQHBv2vDtGfYNYrIIMIV9 1le8HgtXRhIu9FOhw1EA/3m1xc4zNcWMslK6RiV5sn4mBVXUQt6Ry2m/Nar+ccFq =DqcL -END PGP SIGNATURE-
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/04/2012 03:56 PM, Ian Stakenvicius wrote: On 04/09/12 01:32 PM, Zac Medico wrote: On 09/04/2012 10:05 AM, Rick Zero_Chaos Farina wrote: I believe the easiest (and honestly most sane) method is to simply have src_fetch in the live classes check for needed deps and die (with a please emerge blah) if deps are not found. Adding something like FDEPEND just seems to be getting way too crazy on the dependency tree AND would require things to build during fetch-only which doesn't make sense. I think it's nicer to have FDEPEND because it makes the deps more complete, so the package manager can bail out when necessary, without even executing src_fetch. In the case of --fetchonly the package manager could simply bail out if the deps are not installed (like how it bails out for --buildpkgonly when the deps aren't installed). Just looking into the future here; would things like archivers or other helpers used by src_unpack move to FDEPEND as well? or would this be limited solely to tools that data transfer? We are talking about things required for src_fetch (the download) so no, things required for src_unpack have no real place in this as far as I am concerned. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJQRl8nAAoJEKXdFCfdEflKoA0P/152JYx2/62tou9kcUCEILkz DXGfO5bY+A1DYrCsVAR9H7C4dzmrULU0JNjUl3nMpGFKXMAvWQbai2uDICMECZVg QCwHVzxyHD9TzdVAyp928dtn/FtnPvaUV8WHIugFuaZZM/mmACsyEYxFqusMXu51 KkMzs1F9TatL3uoAMuvT0KLiMWHmxw6OrmMrqRX1T7gP4AIy7cD5h8v7DOzWORLQ x12/HGAA2HLDgfhwrNymRs+OIqSLXE54DmAfvOglB6Jfw/I/N6HoY1PQ1vZaW7jU P+t2l7VbRsINmHUk+tbl+7A5F2XBE5GphSh8oIN7yuToSwtbstlGFmTGhb/V0gpM HUpI5v/6B40fc+DgEuAMAZjhUBT+1YIbIO9WVUaXN/I9JBA22FBZSm4qu59qWg/U DX+Jhul9M+A5QPdNBios0K5BYxFXamx/i52w/361NdLV2Onn1zHQT95oC7RKOGJG fZu6PhtkW7Z/CaYSja7TKa8H5pl/+Bj5Mgz7cGlwz41AP4Q6+omfnxlLf+iJME07 rBfKe4RuhZuUXamu+SfW7OdNY8Zqm9skSGw1HbHFVHDQIXKtt8Pv9tr1m364h+W9 r6SUE1xdJaCbtzh2/ajpu9EIVLoVuW/NEFvQIMd7wn3R2PTKgNHOnkLB6c8j/qaa nWe+jCpIt2uSd2Syceqh =4kd2 -END PGP SIGNATURE-
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Tue, 04 Sep 2012 15:56:54 -0400 Ian Stakenvicius a...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 04/09/12 01:32 PM, Zac Medico wrote: On 09/04/2012 10:05 AM, Rick Zero_Chaos Farina wrote: I believe the easiest (and honestly most sane) method is to simply have src_fetch in the live classes check for needed deps and die (with a please emerge blah) if deps are not found. Adding something like FDEPEND just seems to be getting way too crazy on the dependency tree AND would require things to build during fetch-only which doesn't make sense. I think it's nicer to have FDEPEND because it makes the deps more complete, so the package manager can bail out when necessary, without even executing src_fetch. In the case of --fetchonly the package manager could simply bail out if the deps are not installed (like how it bails out for --buildpkgonly when the deps aren't installed). Just looking into the future here; would things like archivers or other helpers used by src_unpack move to FDEPEND as well? or would this be limited solely to tools that data transfer? src_fetch() only. Archivers belong to src_unpack(), which belongs to the main phase function cycle - DEPEND. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
Just looking into the future here; would things like archivers or other helpers used by src_unpack move to FDEPEND as well? or would this be limited solely to tools that data transfer? We should keep the data transfer and the unpack phase clearly separated. So, this would best really be for data transfer only. -- Andreas K. Huettel Gentoo Linux developer kde, sci, arm, tex, printing signature.asc Description: This is a digitally signed message part.
