Le mar. 28 janv. 2025 à 09:27, Christian Lindeberg via
lists.openembedded.org <christian.lindeberg=axis....@lists.openembedded.org>
a écrit :
>
> On 28/01/2025 08:20, Stefan Herbrechtsmeier wrote:
> > Am 27.01.2025 um 23:59 schrieb Martin Jansa:
> >> On Mon, Jan 27, 2025 at 11:11 PM Mark Asselstine via
> >> lists.openembedded.org
> >> <mark.asselstine=windriver....@lists.openembedded.org> wrote:
> >>> On 1/27/2025 4:11 PM, Richard Purdie wrote:
> >>>> On Mon, 2025-01-27 at 15:26 -0500, Mark Asselstine via
> lists.openembedded.org wrote:
> >>>>> On 1/27/2025 8:12 AM, Christian Lindeberg via lists.openembedded.org
> wrote:
> >>>>>> On 27/01/2025 13:22, Stefan Herbrechtsmeier via
> lists.openembedded.org
> >>>>>> wrote:
> >>>>>>> Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via
> >>>>>>> lists.openembedded.org:
> >>>>>>>> Am 27.01.2025 um 10:53 schrieb Richard Purdie:
> >>>>>>>>> On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier wrote:
> >>>>>>>>>> Am 25.01.2025 um 14:17 schrieb Richard Purdie:
> >>>>>>>>>>> The gomod fetcher is totally broken for mirroring
> unfortunately.
> >>>>>>>>>>> It is
> >>>>>>>>>>> used for the crucible recipe in meta-oe and it is one of the
> reasons
> >>>>>>>>>>> the autobuilder mirroring test for meta-oe never passes.
> >>>>>>>>>>>
> >>>>>>>>>>> The issue is that it downloads files into subdirs of DL_DIR
> but when
> >>>>>>>>>>> mirroring, that translation is lost. The files are there on the
> >>>>>>>>>>> mirror,
> >>>>>>>>>>> the code just doesn't have any way to access them.
> >>>>>>>>>> Does this mean the downloadfilename doesn't support sub
> directories or
> >>>>>>>>>> is the rewrite of the URI a problem?
> >>>>>>>>> The rewrite is hard. The syntax doesn't support taking a path
> component
> >>>>>>>>> and injecting it into a parameter.
> >>>>>>>> I assume the problem is not the path but the wrong assumption that
> >>>>>>>> the downloadfilename is used for the replace of the mirror url.
> >>>>>>>>
> >>>>>>>>>> Is the npm fetcher broken too?
> >>>>>>>>> I don't know.
> >>>>>>>> The npm fetcher use a npm2 sub folder in its downloadfilename.
> >>>>>>>>
> >>>>>>>>>>> I don't know what to do about this. The source mirroring for
> meta-oe
> >>>>>>>>>>> has been broken for months. I've tried to fix a couple of the
> issues
> >>>>>>>>>>> but this one is systemic and not easy to fix.
> >>>>>>>>>>>
> >>>>>>>>>>> I'd note that the sanity test in sanity.bbclass doesn't
> recognise the
> >>>>>>>>>>> gomod and godmogit fetchers anyway. I did try a few different
> urls
> >>>>>>>>>>> but
> >>>>>>>>>>> due to the weird way gomod subclasses wget, the normal mirror
> url
> >>>>>>>>>>> manipulation techniques don't work.
> >>>>>>>>>>>
> >>>>>>>>>>> What do I do?
> >>>>>>>>>>>
> >>>>>>>>>>> I could drop meta-oe from source mirroring.
> >>>>>>>>>>>
> >>>>>>>>>>> I could ask Khem to drop the recipe.
> >>>>>>>>>>>
> >>>>>>>>>>> Neither seem like nice things to do but I don't want to do
> "partial"
> >>>>>>>>>>> mirrors. Explaining to people that it may or may not work looks
> >>>>>>>>>>> really
> >>>>>>>>>>> bad and I'm not doing it.
> >>>>>>>>>>>
> >>>>>>>>>>> Is there anyone willing to help fix this? Only stopping it
> running at
> >>>>>>>>>>> all is really within my own control.
> >>>>>>>>>> I can look into it. Would it be okay to init the urldata before
> >>>>>>>>>> resolve
> >>>>>>>>>> the mirror or should the mirror use the gomod scheme?
> >>>>>>>>> Currently the mirror being passed the modified url which makes
> it a
> >>>>>>>>> challenge as it appears as a normal http url.
> >>>>>>>>>
> >>>>>>>>> I posted my workaround that made it work but that has several
> issues.
> >>>>>>>>> It needs a new parameter to the wget fetcher, which might be
> useful
> >>>>>>>>> elsewhere, not sure. It also requires hardcoding the go proxy
> url into
> >>>>>>>>> the PREMIRROR which won't work for any other go proxy servers.
> >>>>>>>> This only works because the downloadfilename match the path of the
> >>>>>>>> uri. I think it would be better to use the downloadfilename inside
> >>>>>>>> the build_mirroruris / uri_replace function, add the
> downloadfilename
> >>>>>>>> to the mirrortarballs or add an additional downloadfiledir
> parameter.
> >>>>>>>>
> >>>>>>>> The following test data for replaceuris in MirrorUriTest should
> >>>>>>>> reproduce the problem:
> >>>>>>>> ("https://somewhere.org/scope/example/1.0.0/
> >>>>>>>>
> example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
> >>>>>>>> "https://.*/.*","http://somewhere2.org/somedir3";)
> >>>>>>>> :"http://somewhere2.org/somedir3/subdir/scope-
> >>>>>>>>
> example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
> >>>>>>>>
> >>>>>>>>
> >>>>>>> The following patch pass the test cases with the additional test
> data
> >>>>>>> above: diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/
> >>>>>>> bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644 --- a/
> >>>>>>> bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/
> >>>>>>> __init__.py @@ -472,7 +472,7 @@ def uri_replace(ud, uri_find,
> >>>>>>> uri_replace, replacements, d, mirrortarball=None): uri_decoded[5]
> = {}
> >>>>>>> uri_find_decoded[5] = {} elif ud.localpath and
> >>>>>>> ud.method.supports_checksum(ud): - basename =
> >>>>>>> os.path.basename(ud.localpath) + basename =
> >>>>>>> ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if
> basename:
> >>>>>>> uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with a
> >>>>>>> slash as a sentinel in case
> >>>>>>>
> >>>>>>> The dot in the lstrip is needed to pass the tests. This need to be
> >>>>>>> resolved in an other way to avoid side-effects.
> >>>>>>>
> >>>>>> Looks like flattening the downloadfilename would solve/workaround
> the
> >>>>>> problem then (replace the slashes with a punctuation character other
> >>>>>> than those valid in a Go module path segment) but I have not had
> time to
> >>>>>> try the additional MirrorUriTest case yet. Is this the desired
> solution
> >>>>>> for wget based fetchers?
> >>>>> The timing of this is uncanny. I actually spent an hour this weekend
> >>>>> looking at an unrelated bug associated with `downloadfilename`
> >>>>> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15126).
> >>>>>
> >>>>> The related test code actually demonstrates some existing confusion
> >>>>> around `downloadfilename` that could result in some very confusing
> >>>>> results. Take a look at
> >>>>> test_fetch_premirror_use_downloadfilename_to_fetch(). The only reason
> >>>>> this test passes is because the specified `downloadfilename`
> >>>>> `bitbake-1.0.tar.gz` actually exists in the premirror. The test
> really
> >>>>> should use a random `downloadfilename` such as `bitcook-1.0.tar.gz`
> >>>>> instead of something that can actually exist.
> >>>>>
> >>>>> If the behavior is left to stand this way it could definitely lead to
> >>>>> head scratching as it would be difficult to understand why checksums
> >>>>> would be failing since what is downloaded is not what is expected.
> >>>>>
> >>>>> My thinking was that `downloadfilename` should only come into play
> when
> >>>>> fetching from DL_DIR and not when fetching from a PREMIRRORS or
> MIRRORS,
> >>>>> but I didn't get that far into reviewing the issue to make any
> conclusions.
> >>>>>
> >>>>> At any rate, I wanted to speak up considering the current discussion
> is
> >>>>> around handling of `downloadfilename` and this might just result in
> even
> >>>>> more confusion on how it should be handled.
> >>>> I'm now travelling so I'm not going to get as much time to think about
> >>>> things as I'd like. I did want to highlight that in theory you can use
> >>>> DL_DIR as a mirror/premirror pretty directly so the naming does need
> to
> >>>> match across them.
> >>> Ya, it is a bit of a philosophical question, when does `A` actually
> >>> become `B`. My rationale for saying that PREMIRRORS and MIRRORS should
> >>> in fact use the original name and not `downloadfilename` is that these
> >>> mirrors may or may not be under the builder's control/ownership,
> whereas
> >>> DL_DIR is. Maybe we need to quantify the PREMIRRORS and MIRRORS as
> being
> >>> prestine or modified?
> >> It might depend on the use-case, but in cases like this:
> >>
> https://git.openembedded.org/meta-openembedded/commit/?id=070b92a3f64e2eee9388d64dc193aaa9d8c115d8
> >> I believe you want downloadfilename to be respected by both DL_DIR and
> >> in PREMIRRORS/MIRRORS, because you don't want to (PRE)MIRROR random
> >> version of UCD.zip and get checksum error whenever changing PV in the
> >> recipe.
> >
> > I think we have different use case to support. The common use case is
> > to use the download folder as a PREMIRRORS and an other use case is to
> > use a private package manager repository or proxy. Only the first use
> > case should use the downloadfilename as path.
> >
> > We could add the value of the downloadfilename parameter to the
> > mirrortarballs class variable and adapt the usage of its value. At the
> > moment the mirrortarball is only used if the scheme is changed.
> > Additional the code removes all parameters. We could use the
> > mirrortarball if the PREMIRROR doesn't use the PATH variable and only
> > remove the parameters if the scheme is changed (except https <-> http).
> >
> > Independent of the PREMIRROR use case we have to decide if the
> > downloadfilename contains a plain filename (basename) or a relative
> > path. At the moment parts of the code assume a plain filename
> > (basename) and other parts put a relative path into the downloadfilename.
> >
>
> In the UCD.zip case only the unpacked content of the zip file matters
> and not
> the name of the file itself (if I'm not completely mistaken)
On that, let me present a failure we periodically have on the mirror test
CI :
you have a recipe downloading an archive with a constant name (no PV inside
the name) : let's say package.zip
on upgrade, archive hash change but what happens on the mirror test is:
* package.zip is not re-downloaded since it is already present on mirror
* when trying to build using only local data, it fails because of hash
mismatch.
On a normal build, this would trigger a redownload and the build would
ultimately work.
But the mirror testing code is here to test build from mirror (and only
from mirror).
I thought about adding a test for "does downloadfilename contains PV?" but
never came around to actually do it (or think harder about if that a good
idea or not).
> but in the
> case of
> the gomod fetcher the "unpackfilename" matters and I didn't find a way to
> specify it independetly of downloadfilename so just flattening the path
> was not
> as simple as I first thought. The relative path is there as the simplest
> mean
> to avoid name conflicts in DL_DIR
>
> Also, I saw that the following test data for replaceuris in MirrorUriTest
> worked fine:
> ("gomod://golang.org/x/net;version=v0.9.0",
> "https://proxy.golang.org/";, "file:///somewhere/go-proxy/")
> :
> "file:///somewhere/go-proxy/golang.org/x/net/%40v/v0.9.0.zip",
>
>
>
>
>
--
Yoann Congal
Smile ECS - Tech expert
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2114):
https://lists.openembedded.org/g/openembedded-architecture/message/2114
Mute This Topic: https://lists.openembedded.org/mt/110806536/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
