Re: [OE-core] File conflicts in DL_DIR

[Phil Blundell]

On Wed, 2016-10-12 at 11:03 -0700, Matt Madison wrote:
> I use ";downloadfilename=${BPN}-${PV}.tar.gz" to deal with this
> problem, and it's worked well for me.

Thanks, that works for me.

p.

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] File conflicts in DL_DIR

[Martin Jansa]

On Wed, Oct 12, 2016 at 08:11:57PM +0100, Burton, Ross wrote:
> On 12 October 2016 at 18:30, Martin Jansa  wrote:
> 
> > Github archives are regenerated from time to time and then the checksums
> > are changed, so 3rd option is to use git fetcher with SRCREV matching the
> > has of v${PV} tag.
> >
> 
> Note that Github maintain that this isn't the case for .tar.gz archives.

I've checked git log of my layers and from last 3 cases where I was
replacing github archive with git SRC_URI it was .tar.gz in two cases and
.zip in one (and they were really failing to fetch because checksums
mismatch).

They might maintain that they don't regenerate them anymore, but if you have
older recipe with checksums which were valid 2 years ago you can still
be surprised by new checksums (I was 3 times in last year by github.com,
last month from few more cases which were still using googlecode.com and
direct release URL stopped working).

Unfortunately this always happens close (or worse shortly after) our
internal release when we switch to different premirror directory and
recipes start failing to fetch.

-- 
Martin 'JaMa' Jansa jabber: martin.ja...@gmail.com


signature.asc
Description: Digital signature
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] File conflicts in DL_DIR

[Burton, Ross]

On 12 October 2016 at 18:30, Martin Jansa  wrote:

> Github archives are regenerated from time to time and then the checksums
> are changed, so 3rd option is to use git fetcher with SRCREV matching the
> has of v${PV} tag.
>

Note that Github maintain that this isn't the case for .tar.gz archives.

Ross
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] File conflicts in DL_DIR

[Matt Madison]

On Wed, Oct 12, 2016 at 10:01 AM, Phil Blundell  wrote:
> I just ran into an annoying problem with two recipes whose upstreams
> both use the same poor naming scheme for their downloadable tarballs
> which causes the sources to collide in DL_DIR.  One of them is eudev,
> which has:
>
> SRC_URI = "https://github.com/gentoo/${BPN}/archive/v${PV}.tar.gz
>
> This shows up as a source checksum error when the second package finds
> what it thinks are its sources in DL_DIR, but of course they are not
> the files that it wanted.
>
> I couldn't immediately find any easy way to work around this
> difficulty.  Setting "localpath=eudev_v{$PV}.tar.gz" in eudev's SRC_URI
> looks superficially like it ought to do the right thing, but although
> this changes the location that the fetcher will search for the
> downloaded file it doesn't actually cause wget to put the file there so
> this doesn't work either.

I use ";downloadfilename=${BPN}-${PV}.tar.gz" to deal with this
problem, and it's worked well for me.

-Matt

>
> I suppose I could define DL_DIR in terms of ${PN}, which would
> presumably fix this problem at the cost of causing all my previous
> downloads to be repeated.  Or I could stop using eudev, or wait for one
> of the two packages to release a new version. :-)  Any better
> suggestions?
>
> thanks
>
> p.
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] File conflicts in DL_DIR

[Christopher Larson]

On Wed, Oct 12, 2016 at 10:01 AM, Phil Blundell  wrote:

> I just ran into an annoying problem with two recipes whose upstreams
> both use the same poor naming scheme for their downloadable tarballs
> which causes the sources to collide in DL_DIR.  One of them is eudev,
> which has:
>
> SRC_URI = "https://github.com/gentoo/${BPN}/archive/v${PV}.tar.gz
>
> This shows up as a source checksum error when the second package finds
> what it thinks are its sources in DL_DIR, but of course they are not
> the files that it wanted.
>
> I couldn't immediately find any easy way to work around this
> difficulty.  Setting "localpath=eudev_v{$PV}.tar.gz" in eudev's SRC_URI
> looks superficially like it ought to do the right thing, but although
> this changes the location that the fetcher will search for the
> downloaded file it doesn't actually cause wget to put the file there so
> this doesn't work either.
>

I think that’s just a bug.. The fetcher should fully obey the url
parameters we support, so we should fix the wget fetcher to properly obey
localpath.

I suppose I could define DL_DIR in terms of ${PN}, which would
> presumably fix this problem at the cost of causing all my previous
> downloads to be repeated.  Or I could stop using eudev, or wait for one
> of the two packages to release a new version. :-)  Any better
> suggestions?
>

One option would be to change DL_DIR, but use PREMIRRORS for compatibility
with existing DL_DIRs, i.e.:
DL_DIR_BASE = “${TOPDIR}/downloads”
DL_DIR = “${DL_DIR_BASE}/${BPN}”
PREMIRRORS_prepend = “file://.*/ file://${DL_DIR_BASE}/\n “

or similar (untested).

Of course, this conflict will occur on a mirror too, so that’s a concern if
you care about mirror population.

Long term, I’d like to see our DL_DIR as a more capable download cache.
Download every file to a unique name based on its checksum, then link those
into a ${BPN}-separated area with their real names. Then add a separate
tool to populate a mirror from DL_DIR.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] File conflicts in DL_DIR

[Martin Jansa]

On Wed, Oct 12, 2016 at 06:01:05PM +0100, Phil Blundell wrote:
> I just ran into an annoying problem with two recipes whose upstreams
> both use the same poor naming scheme for their downloadable tarballs
> which causes the sources to collide in DL_DIR.  One of them is eudev,
> which has:
> 
> SRC_URI = "https://github.com/gentoo/${BPN}/archive/v${PV}.tar.gz

Github archives are regenerated from time to time and then the checksums
are changed, so 3rd option is to use git fetcher with SRCREV matching the
has of v${PV} tag.

> This shows up as a source checksum error when the second package finds
> what it thinks are its sources in DL_DIR, but of course they are not
> the files that it wanted.
> 
> I couldn't immediately find any easy way to work around this
> difficulty.  Setting "localpath=eudev_v{$PV}.tar.gz" in eudev's SRC_URI
> looks superficially like it ought to do the right thing, but although
> this changes the location that the fetcher will search for the
> downloaded file it doesn't actually cause wget to put the file there so
> this doesn't work either.
> 
> I suppose I could define DL_DIR in terms of ${PN}, which would
> presumably fix this problem at the cost of causing all my previous
> downloads to be repeated.  Or I could stop using eudev, or wait for one
> of the two packages to release a new version. :-)  Any better
> suggestions?
> 
> thanks
> 
> p.
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa jabber: martin.ja...@gmail.com


signature.asc
Description: Digital signature
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core