Am 31.01.2025 um 16:37 schrieb Peter Kjellerstedt:
-----Original Message-----
From: openembedded-architecture@lists.openembedded.org <openembedded-
architect...@lists.openembedded.org> On Behalf Of Mark Asselstine via
lists.openembedded.org
Sent: den 29 januari 2025 16:28
To: stefan.herbrechtsmeier-...@weidmueller.com; openembedded-
architect...@lists.openembedded.org
Subject: Re: [Openembedded-architecture] gomod fetcher totally broken for
mirroring
On 1/29/2025 4:18 AM, Stefan Herbrechtsmeier via lists.openembedded.org
wrote:
Am 28.01.2025 um 15:47 schrieb Mark Asselstine via
lists.openembedded.org:
On 1/28/2025 2:20 AM, Stefan Herbrechtsmeier via
lists.openembedded.org wrote:
Am 27.01.2025 um 23:59 schrieb Martin Jansa:
On Mon, Jan 27, 2025 at 11:11 PM Mark Asselstine via
lists.openembedded.org
<mark.asselstine=windriver....@lists.openembedded.org> wrote:
On 1/27/2025 4:11 PM, Richard Purdie wrote:
On Mon, 2025-01-27 at 15:26 -0500, Mark Asselstine via
lists.openembedded.org wrote:
On 1/27/2025 8:12 AM, Christian Lindeberg via
lists.openembedded.org wrote:
On 27/01/2025 13:22, Stefan Herbrechtsmeier via
lists.openembedded.org
wrote:
Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via
lists.openembedded.org:
Am 27.01.2025 um 10:53 schrieb Richard Purdie:
On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier
wrote:
Am 25.01.2025 um 14:17 schrieb Richard Purdie:
The gomod fetcher is totally broken for mirroring
unfortunately.
It is
used for the crucible recipe in meta-oe and it is one of
the reasons
the autobuilder mirroring test for meta-oe never passes.
The issue is that it downloads files into subdirs of DL_DIR
but when
mirroring, that translation is lost. The files are there on
the
mirror,
the code just doesn't have any way to access them.
Does this mean the downloadfilename doesn't support sub
directories or
is the rewrite of the URI a problem?
The rewrite is hard. The syntax doesn't support taking a path
component
and injecting it into a parameter.
I assume the problem is not the path but the wrong assumption
that
the downloadfilename is used for the replace of the mirror url.
Is the npm fetcher broken too?
I don't know.
The npm fetcher use a npm2 sub folder in its downloadfilename.
I don't know what to do about this. The source mirroring
for meta-oe
has been broken for months. I've tried to fix a couple of
the issues
but this one is systemic and not easy to fix.
I'd note that the sanity test in sanity.bbclass doesn't
recognise the
gomod and godmogit fetchers anyway. I did try a few
different urls
but
due to the weird way gomod subclasses wget, the normal
mirror url
manipulation techniques don't work.
What do I do?
I could drop meta-oe from source mirroring.
I could ask Khem to drop the recipe.
Neither seem like nice things to do but I don't want to do
"partial"
mirrors. Explaining to people that it may or may not work
looks
really
bad and I'm not doing it.
Is there anyone willing to help fix this? Only stopping it
running at
all is really within my own control.
I can look into it. Would it be okay to init the urldata
before
resolve
the mirror or should the mirror use the gomod scheme?
Currently the mirror being passed the modified url which
makes it a
challenge as it appears as a normal http url.
I posted my workaround that made it work but that has several
issues.
It needs a new parameter to the wget fetcher, which might be
useful
elsewhere, not sure. It also requires hardcoding the go proxy
url into
the PREMIRROR which won't work for any other go proxy servers.
This only works because the downloadfilename match the path of
the
uri. I think it would be better to use the downloadfilename
inside
the build_mirroruris / uri_replace function, add the
downloadfilename
to the mirrortarballs or add an additional downloadfiledir
parameter.
The following test data for replaceuris in MirrorUriTest should
reproduce the problem:
("https://somewhere.org/scope/example/1.0.0/
example-1.0.0.tgz;downloadfilename=subdir/scope-
example-1.0.0.tgz",
"https://.*/.*","http://somewhere2.org/somedir3";)
:"http://somewhere2.org/somedir3/subdir/scope-
example-1.0.0.tgz;downloadfilename=subdir/scope-
example-1.0.0.tgz",
The following patch pass the test cases with the additional
test data
above: diff --git a/bitbake/lib/bb/fetch2/__init__.py b/
bitbake/lib/
bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644 --- a/
bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/
__init__.py @@ -472,7 +472,7 @@ def uri_replace(ud, uri_find,
uri_replace, replacements, d, mirrortarball=None):
uri_decoded[5] = {}
uri_find_decoded[5] = {} elif ud.localpath and
ud.method.supports_checksum(ud): - basename =
os.path.basename(ud.localpath) + basename =
ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if
basename:
uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with
a
slash as a sentinel in case
The dot in the lstrip is needed to pass the tests. This need to
be
resolved in an other way to avoid side-effects.
Looks like flattening the downloadfilename would solve/
workaround the
problem then (replace the slashes with a punctuation character
other
than those valid in a Go module path segment) but I have not had
time to
try the additional MirrorUriTest case yet. Is this the desired
solution
for wget based fetchers?
The timing of this is uncanny. I actually spent an hour this
weekend
looking at an unrelated bug associated with `downloadfilename`
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=15126).
The related test code actually demonstrates some existing
confusion
around `downloadfilename` that could result in some very confusing
results. Take a look at
test_fetch_premirror_use_downloadfilename_to_fetch(). The only
reason
this test passes is because the specified `downloadfilename`
`bitbake-1.0.tar.gz` actually exists in the premirror. The test
really
should use a random `downloadfilename` such as `bitcook-
1.0.tar.gz`
instead of something that can actually exist.
If the behavior is left to stand this way it could definitely
lead to
head scratching as it would be difficult to understand why
checksums
would be failing since what is downloaded is not what is expected.
My thinking was that `downloadfilename` should only come into
play when
fetching from DL_DIR and not when fetching from a PREMIRRORS or
MIRRORS,
but I didn't get that far into reviewing the issue to make any
conclusions.
At any rate, I wanted to speak up considering the current
discussion is
around handling of `downloadfilename` and this might just result
in even
more confusion on how it should be handled.
I'm now travelling so I'm not going to get as much time to think
about
things as I'd like. I did want to highlight that in theory you can
use
DL_DIR as a mirror/premirror pretty directly so the naming does
need to
match across them.
Ya, it is a bit of a philosophical question, when does `A` actually
become `B`. My rationale for saying that PREMIRRORS and MIRRORS
should
in fact use the original name and not `downloadfilename` is that
these
mirrors may or may not be under the builder's control/ownership,
whereas
DL_DIR is. Maybe we need to quantify the PREMIRRORS and MIRRORS as
being
prestine or modified?
It might depend on the use-case, but in cases like this:
https://git.openembedded.org/meta-openembedded/commit/?
id=070b92a3f64e2eee9388d64dc193aaa9d8c115d8
I believe you want downloadfilename to be respected by both DL_DIR
and
in PREMIRRORS/MIRRORS, because you don't want to (PRE)MIRROR random
version of UCD.zip and get checksum error whenever changing PV in the
recipe.
I think we have different use case to support. The common use case is
to use the download folder as a PREMIRRORS and an other use case is
to use a private package manager repository or proxy. Only the first
use case should use the downloadfilename as path.
Agreed.
We could add the value of the downloadfilename parameter to the
mirrortarballs class variable and adapt the usage of its value. At
the moment the mirrortarball is only used if the scheme is changed.
Additional the code removes all parameters. We could use the
mirrortarball if the PREMIRROR doesn't use the PATH variable and only
remove the parameters if the scheme is changed (except https <->
http).
Independent of the PREMIRROR use case we have to decide if the
downloadfilename contains a plain filename (basename) or a relative
path. At the moment parts of the code assume a plain filename
(basename) and other parts put a relative path into the
downloadfilename.
If we review the documentation I would argue that `downloadfilename`
should be in effect for DL_DIR and the usecase you express above,
where DL_DIR is essentially turned into a PREMIRRORS. But MIRRORS
should not use `downloadfilename`. Here's the documentation I am
referring to:
There are two kinds of mirrors: pre-mirrors and regular mirrors.
The
PREMIRRORS and MIRRORS variables point to these, respectively.
BitBake checks pre-mirrors before looking upstream for any source
files. Pre-mirrors are appropriate when you have a shared directory
that is not a directory defined by the DL_DIR variable. A Pre-
mirror
typically points to a shared directory that is local to your
organization.
NOTE the term `regular mirrors` and `upstream`. ie. MIRRORS are not a
Yocto Project entity, they are truly a backup.
Technically the different between pre-mirrors and mirrors is that the
pre-mirrors are try before and the mirrors after upstream.
Understood.
A good example of this would be PyPi and a PyPi mirror (something that
isn't out of the ordinary to setup).
This is a good example for a pre-mirror with mirror upstream. A private
PyPi mirror should be used before the normal upstream and not only if
the upstream failed.
But this still leaves things open. If I were to create a private PyPi
mirror I would most likely do so using one of the various tools out
there to create it. In that case the names used would be the same as in
PyPi and thus I wouldn't want downloadfilename to be in play for the
fetch.
MarkA
Doesn't this mean that what we really want is something like a
DL_DIR_MIRROR (or PRE_DL_DIR?), which has the same semantics as DL_DIR
(basically what PREMIRROR was originally intended for)? And then
PREMIRROR should behave like upstream and MIRROR with regards to, e.g.,
downloadfilename.
Alternative we could add a DOWNLOADFILENAME replacement and make it
explicit in the PREMIRROR variable:
https?://.*/.*
http://downloads.yoctoproject.org/mirror/sources/DOWNLOADFILENAME
The sstate mirror only works because it uses the PATH in the path and
its value match the downloadfilename parameter:
file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH
With the DOWNLOADFILENAME replacement it could be:
file://.* http://sstate.yoctoproject.org/all/DOWNLOADFILENAME
Without the DOWNLOADFILENAME the mirror will keep the original PATH.
This allows the mirror to override the last part of the path which is
impossible with the current implementation.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2123):
https://lists.openembedded.org/g/openembedded-architecture/message/2123
Mute This Topic: https://lists.openembedded.org/mt/110806536/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
