Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via
lists.openembedded.org:
Am 27.01.2025 um 10:53 schrieb Richard Purdie:
On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier wrote:
Am 25.01.2025 um 14:17 schrieb Richard Purdie:
The gomod fetcher is totally broken for mirroring unfortunately. It is
used for the crucible recipe in meta-oe and it is one of the reasons
the autobuilder mirroring test for meta-oe never passes.
The issue is that it downloads files into subdirs of DL_DIR but when
mirroring, that translation is lost. The files are there on the mirror,
the code just doesn't have any way to access them.
Does this mean the downloadfilename doesn't support sub directories or
is the rewrite of the URI a problem?
The rewrite is hard. The syntax doesn't support taking a path component
and injecting it into a parameter.
I assume the problem is not the path but the wrong assumption that the
downloadfilename is used for the replace of the mirror url.
Is the npm fetcher broken too?
I don't know.
The npm fetcher use a npm2 sub folder in its downloadfilename.
I don't know what to do about this. The source mirroring for meta-oe
has been broken for months. I've tried to fix a couple of the issues
but this one is systemic and not easy to fix.
I'd note that the sanity test in sanity.bbclass doesn't recognise the
gomod and godmogit fetchers anyway. I did try a few different urls but
due to the weird way gomod subclasses wget, the normal mirror url
manipulation techniques don't work.
What do I do?
I could drop meta-oe from source mirroring.
I could ask Khem to drop the recipe.
Neither seem like nice things to do but I don't want to do "partial"
mirrors. Explaining to people that it may or may not work looks really
bad and I'm not doing it.
Is there anyone willing to help fix this? Only stopping it running at
all is really within my own control.
I can look into it. Would it be okay to init the urldata before resolve
the mirror or should the mirror use the gomod scheme?
Currently the mirror being passed the modified url which makes it a
challenge as it appears as a normal http url.
I posted my workaround that made it work but that has several issues.
It needs a new parameter to the wget fetcher, which might be useful
elsewhere, not sure. It also requires hardcoding the go proxy url into
the PREMIRROR which won't work for any other go proxy servers.
This only works because the downloadfilename match the path of the
uri. I think it would be better to use the downloadfilename inside the
build_mirroruris / uri_replace function, add the downloadfilename to
the mirrortarballs or add an additional downloadfiledir parameter.
The following test data for replaceuris in MirrorUriTest should
reproduce the problem:
("https://somewhere.org/scope/example/1.0.0/example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz";,
"https://.*/.*";, "http://somewhere2.org/somedir3";)
:
"http://somewhere2.org/somedir3/subdir/scope-example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz";,
The following patch pass the test cases with the additional test data
above: diff --git a/bitbake/lib/bb/fetch2/__init__.py
b/bitbake/lib/bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644
--- a/bitbake/lib/bb/fetch2/__init__.py +++
b/bitbake/lib/bb/fetch2/__init__.py @@ -472,7 +472,7 @@ def
uri_replace(ud, uri_find, uri_replace, replacements, d,
mirrortarball=None): uri_decoded[5] = {} uri_find_decoded[5] = {} elif
ud.localpath and ud.method.supports_checksum(ud): - basename =
os.path.basename(ud.localpath) + basename =
ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if basename:
uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with a slash
as a sentinel in case
The dot in the lstrip is needed to pass the tests. This need to be
resolved in an other way to avoid side-effects.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2106):
https://lists.openembedded.org/g/openembedded-architecture/message/2106
Mute This Topic: https://lists.openembedded.org/mt/110806536/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
