On 1/27/2025 8:12 AM, Christian Lindeberg via lists.openembedded.org wrote:
On 27/01/2025 13:22, Stefan Herbrechtsmeier via lists.openembedded.org
wrote:
Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via
lists.openembedded.org:
Am 27.01.2025 um 10:53 schrieb Richard Purdie:
On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier wrote:
Am 25.01.2025 um 14:17 schrieb Richard Purdie:
The gomod fetcher is totally broken for mirroring unfortunately.
It is
used for the crucible recipe in meta-oe and it is one of the reasons
the autobuilder mirroring test for meta-oe never passes.
The issue is that it downloads files into subdirs of DL_DIR but when
mirroring, that translation is lost. The files are there on the
mirror,
the code just doesn't have any way to access them.
Does this mean the downloadfilename doesn't support sub directories or
is the rewrite of the URI a problem?
The rewrite is hard. The syntax doesn't support taking a path component
and injecting it into a parameter.
I assume the problem is not the path but the wrong assumption that
the downloadfilename is used for the replace of the mirror url.
Is the npm fetcher broken too?
I don't know.
The npm fetcher use a npm2 sub folder in its downloadfilename.
I don't know what to do about this. The source mirroring for meta-oe
has been broken for months. I've tried to fix a couple of the issues
but this one is systemic and not easy to fix.
I'd note that the sanity test in sanity.bbclass doesn't recognise the
gomod and godmogit fetchers anyway. I did try a few different urls
but
due to the weird way gomod subclasses wget, the normal mirror url
manipulation techniques don't work.
What do I do?
I could drop meta-oe from source mirroring.
I could ask Khem to drop the recipe.
Neither seem like nice things to do but I don't want to do "partial"
mirrors. Explaining to people that it may or may not work looks
really
bad and I'm not doing it.
Is there anyone willing to help fix this? Only stopping it running at
all is really within my own control.
I can look into it. Would it be okay to init the urldata before
resolve
the mirror or should the mirror use the gomod scheme?
Currently the mirror being passed the modified url which makes it a
challenge as it appears as a normal http url.
I posted my workaround that made it work but that has several issues.
It needs a new parameter to the wget fetcher, which might be useful
elsewhere, not sure. It also requires hardcoding the go proxy url into
the PREMIRROR which won't work for any other go proxy servers.
This only works because the downloadfilename match the path of the
uri. I think it would be better to use the downloadfilename inside
the build_mirroruris / uri_replace function, add the downloadfilename
to the mirrortarballs or add an additional downloadfiledir parameter.
The following test data for replaceuris in MirrorUriTest should
reproduce the problem:
("https://somewhere.org/scope/example/1.0.0/
example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
"https://.*/.*";, "http://somewhere2.org/somedir3";)
: "http://somewhere2.org/somedir3/subdir/scope-
example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
The following patch pass the test cases with the additional test data
above: diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/
bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644 --- a/
bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/
__init__.py @@ -472,7 +472,7 @@ def uri_replace(ud, uri_find,
uri_replace, replacements, d, mirrortarball=None): uri_decoded[5] = {}
uri_find_decoded[5] = {} elif ud.localpath and
ud.method.supports_checksum(ud): - basename =
os.path.basename(ud.localpath) + basename =
ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if basename:
uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with a
slash as a sentinel in case
The dot in the lstrip is needed to pass the tests. This need to be
resolved in an other way to avoid side-effects.
Looks like flattening the downloadfilename would solve/workaround the
problem then (replace the slashes with a punctuation character other
than those valid in a Go module path segment) but I have not had time to
try the additional MirrorUriTest case yet. Is this the desired solution
for wget based fetchers?
The timing of this is uncanny. I actually spent an hour this weekend
looking at an unrelated bug associated with `downloadfilename`
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=15126).
The related test code actually demonstrates some existing confusion
around `downloadfilename` that could result in some very confusing
results. Take a look at
test_fetch_premirror_use_downloadfilename_to_fetch(). The only reason
this test passes is because the specified `downloadfilename`
`bitbake-1.0.tar.gz` actually exists in the premirror. The test really
should use a random `downloadfilename` such as `bitcook-1.0.tar.gz`
instead of something that can actually exist.
If the behavior is left to stand this way it could definitely lead to
head scratching as it would be difficult to understand why checksums
would be failing since what is downloaded is not what is expected.
My thinking was that `downloadfilename` should only come into play when
fetching from DL_DIR and not when fetching from a PREMIRRORS or MIRRORS,
but I didn't get that far into reviewing the issue to make any conclusions.
At any rate, I wanted to speak up considering the current discussion is
around handling of `downloadfilename` and this might just result in even
more confusion on how it should be handled.
MarkA
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2108):
https://lists.openembedded.org/g/openembedded-architecture/message/2108
Mute This Topic: https://lists.openembedded.org/mt/110806536/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
