On Mon, Jan 27, 2025 at 11:11 PM Mark Asselstine via
lists.openembedded.org
<mark.asselstine=windriver....@lists.openembedded.org> wrote:
>
>
>
> On 1/27/2025 4:11 PM, Richard Purdie wrote:
> > On Mon, 2025-01-27 at 15:26 -0500, Mark Asselstine via
> > lists.openembedded.org wrote:
> >>
> >>
> >> On 1/27/2025 8:12 AM, Christian Lindeberg via lists.openembedded.org wrote:
> >>>
> >>>
> >>> On 27/01/2025 13:22, Stefan Herbrechtsmeier via lists.openembedded.org
> >>> wrote:
> >>>> Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via
> >>>> lists.openembedded.org:
> >>>>> Am 27.01.2025 um 10:53 schrieb Richard Purdie:
> >>>>>> On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier wrote:
> >>>>>>> Am 25.01.2025 um 14:17 schrieb Richard Purdie:
> >>>>>>>> The gomod fetcher is totally broken for mirroring unfortunately.
> >>>>>>>> It is
> >>>>>>>> used for the crucible recipe in meta-oe and it is one of the reasons
> >>>>>>>> the autobuilder mirroring test for meta-oe never passes.
> >>>>>>>>
> >>>>>>>> The issue is that it downloads files into subdirs of DL_DIR but when
> >>>>>>>> mirroring, that translation is lost. The files are there on the
> >>>>>>>> mirror,
> >>>>>>>> the code just doesn't have any way to access them.
> >>>>>>> Does this mean the downloadfilename doesn't support sub directories or
> >>>>>>> is the rewrite of the URI a problem?
> >>>>>> The rewrite is hard. The syntax doesn't support taking a path component
> >>>>>> and injecting it into a parameter.
> >>>>>
> >>>>> I assume the problem is not the path but the wrong assumption that
> >>>>> the downloadfilename is used for the replace of the mirror url.
> >>>>>
> >>>>>>> Is the npm fetcher broken too?
> >>>>>> I don't know.
> >>>>>
> >>>>> The npm fetcher use a npm2 sub folder in its downloadfilename.
> >>>>>
> >>>>>>>> I don't know what to do about this. The source mirroring for meta-oe
> >>>>>>>> has been broken for months. I've tried to fix a couple of the issues
> >>>>>>>> but this one is systemic and not easy to fix.
> >>>>>>>>
> >>>>>>>> I'd note that the sanity test in sanity.bbclass doesn't recognise the
> >>>>>>>> gomod and godmogit fetchers anyway. I did try a few different urls
> >>>>>>>> but
> >>>>>>>> due to the weird way gomod subclasses wget, the normal mirror url
> >>>>>>>> manipulation techniques don't work.
> >>>>>>>>
> >>>>>>>> What do I do?
> >>>>>>>>
> >>>>>>>> I could drop meta-oe from source mirroring.
> >>>>>>>>
> >>>>>>>> I could ask Khem to drop the recipe.
> >>>>>>>>
> >>>>>>>> Neither seem like nice things to do but I don't want to do "partial"
> >>>>>>>> mirrors. Explaining to people that it may or may not work looks
> >>>>>>>> really
> >>>>>>>> bad and I'm not doing it.
> >>>>>>>>
> >>>>>>>> Is there anyone willing to help fix this? Only stopping it running at
> >>>>>>>> all is really within my own control.
> >>>>>>> I can look into it. Would it be okay to init the urldata before
> >>>>>>> resolve
> >>>>>>> the mirror or should the mirror use the gomod scheme?
> >>>>>> Currently the mirror being passed the modified url which makes it a
> >>>>>> challenge as it appears as a normal http url.
> >>>>>>
> >>>>>> I posted my workaround that made it work but that has several issues.
> >>>>>> It needs a new parameter to the wget fetcher, which might be useful
> >>>>>> elsewhere, not sure. It also requires hardcoding the go proxy url into
> >>>>>> the PREMIRROR which won't work for any other go proxy servers.
> >>>>>
> >>>>> This only works because the downloadfilename match the path of the
> >>>>> uri. I think it would be better to use the downloadfilename inside
> >>>>> the build_mirroruris / uri_replace function, add the downloadfilename
> >>>>> to the mirrortarballs or add an additional downloadfiledir parameter.
> >>>>>
> >>>>> The following test data for replaceuris in MirrorUriTest should
> >>>>> reproduce the problem:
> >>>>> ("https://somewhere.org/scope/example/1.0.0/
> >>>>> example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
> >>>>> "https://.*/.*";, "http://somewhere2.org/somedir3";)
> >>>>> : "http://somewhere2.org/somedir3/subdir/scope-
> >>>>> example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
> >>>>>
> >>>>>
> >>>> The following patch pass the test cases with the additional test data
> >>>> above: diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/
> >>>> bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644 --- a/
> >>>> bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/
> >>>> __init__.py @@ -472,7 +472,7 @@ def uri_replace(ud, uri_find,
> >>>> uri_replace, replacements, d, mirrortarball=None): uri_decoded[5] = {}
> >>>> uri_find_decoded[5] = {} elif ud.localpath and
> >>>> ud.method.supports_checksum(ud): - basename =
> >>>> os.path.basename(ud.localpath) + basename =
> >>>> ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if basename:
> >>>> uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with a
> >>>> slash as a sentinel in case
> >>>>
> >>>> The dot in the lstrip is needed to pass the tests. This need to be
> >>>> resolved in an other way to avoid side-effects.
> >>>>
> >>> Looks like flattening the downloadfilename would solve/workaround the
> >>> problem then (replace the slashes with a punctuation character other
> >>> than those valid in a Go module path segment) but I have not had time to
> >>> try the additional MirrorUriTest case yet. Is this the desired solution
> >>> for wget based fetchers?
> >>
> >> The timing of this is uncanny. I actually spent an hour this weekend
> >> looking at an unrelated bug associated with `downloadfilename`
> >> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15126).
> >>
> >> The related test code actually demonstrates some existing confusion
> >> around `downloadfilename` that could result in some very confusing
> >> results. Take a look at
> >> test_fetch_premirror_use_downloadfilename_to_fetch(). The only reason
> >> this test passes is because the specified `downloadfilename`
> >> `bitbake-1.0.tar.gz` actually exists in the premirror. The test really
> >> should use a random `downloadfilename` such as `bitcook-1.0.tar.gz`
> >> instead of something that can actually exist.
> >>
> >> If the behavior is left to stand this way it could definitely lead to
> >> head scratching as it would be difficult to understand why checksums
> >> would be failing since what is downloaded is not what is expected.
> >>
> >> My thinking was that `downloadfilename` should only come into play when
> >> fetching from DL_DIR and not when fetching from a PREMIRRORS or MIRRORS,
> >> but I didn't get that far into reviewing the issue to make any conclusions.
> >>
> >> At any rate, I wanted to speak up considering the current discussion is
> >> around handling of `downloadfilename` and this might just result in even
> >> more confusion on how it should be handled.
> >
> > I'm now travelling so I'm not going to get as much time to think about
> > things as I'd like. I did want to highlight that in theory you can use
> > DL_DIR as a mirror/premirror pretty directly so the naming does need to
> > match across them.
>
> Ya, it is a bit of a philosophical question, when does `A` actually
> become `B`. My rationale for saying that PREMIRRORS and MIRRORS should
> in fact use the original name and not `downloadfilename` is that these
> mirrors may or may not be under the builder's control/ownership, whereas
> DL_DIR is. Maybe we need to quantify the PREMIRRORS and MIRRORS as being
> prestine or modified?
It might depend on the use-case, but in cases like this:
https://git.openembedded.org/meta-openembedded/commit/?id=070b92a3f64e2eee9388d64dc193aaa9d8c115d8
I believe you want downloadfilename to be respected by both DL_DIR and
in PREMIRRORS/MIRRORS, because you don't want to (PRE)MIRROR random
version of UCD.zip and get checksum error whenever changing PV in the
recipe.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2111):
https://lists.openembedded.org/g/openembedded-architecture/message/2111
Mute This Topic: https://lists.openembedded.org/mt/110806536/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
