On Thu, 2013-07-11 at 09:51 -0400, Zdenek Pavlas wrote:
> > Why change _commonRetrieveDataMD_done to be revertable, and
> > apply/retry/etc. there instead of applying in the download phase.
>
> I'd have to apply deltas and checksum the result from checkfunc callback..
> and it blocks both the progress display and the multi-downloader scheduler.
> Checksumming is probably the biggest blocker here, and since open files
> are about 3-5 times larger, it might be bad.
Yeh, but we have to block and do this somewhere ... it just seems like
redoing all the logic so that the bit that couldn't fail can now fail
seems bad.
> Also, when we download and checksum delta, then apply it, and the result
> does not match, we should not try other mirror. Now there's UrlGrabError(-3)
> that does this, but still we can't change the relative URL and retry..
>
> IMO it has to be done in 3 steps:
> 1) get & checksum delta
> 2) unzip & apply & checksum the result
> 3) full DL if 2) fails.
Maybe have some way to do this and go back into
_commonRetrieveDataMD_list on some failures?
This is basically the drpm problem again ... and even the first
versions of that didn't just download all the drpms and then stop
everything and generate the rpms.
> > And, again ... it's going to be a lot of debugging pain if the output
> > of the deltas don't correspond to anything verifiable from the repo.
>
> I do (open) checksum of the result.
Yeh, I know. So it's _much_ better than just assuming:
old(presumably good)+new delta(checksummed) = good
...but the result still isn't directly in repomd.xml.
> > At _best_ every time we have a weird bug we'll have to question if it's the
> > metadata and there will be no way to verify anything that happened.
>
> The output XML files are checked against repomd.xml on each run,
> (using the xattr checksum caching)..
>
> Still, local rebuild might produce slightly different .sqlite file
> than the one on the server.. should we check this, too?
No, if we have something that we can directly check against the server
that's what we have now.
> > To expand here, we pass the open_checksum ... so "all" we need to do is
> > compress it in the same way as upstream, and make sure it matches.
>
> That does not make sense.. Openchecksum has the same strength and comes
> from the same domain as closedchecksum.. why bother compressing?
So currently we have:
1. metalink.xml (SSL)
2. \_ repomd.xml (checksummed from metalink)
3. \_ primary.xml.blah (checksummed from repomd.xml)
4. \_ gen/primary.xml (generated from #3)
(also satisfies open checksum)
5. \_ gen/primary.xml.gz.sqlite (generated from #4)
...everything in gen/ is local content that has been generated from
verified network content that directly flowed from the initial SSL
connection and can be copied/shared by other users/whatever.
This would change it to:
1. metalink.xml (SSL)
2. \_ repomd.xml (checksummed from metalink)
3. \_ ????
4. \_ gen/primary.xml (generated from #3, gg)
(also satisfies open checksum)
5. \_ gen/primary.xml.gz.sqlite (generated from #4)
...now we can't delete bits of stuff in gen/ ... anyone wanting to
copy/share our data needs to work out what to replace ???? with.
_______________________________________________
Yum-devel mailing list
Yum-devel@lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/yum-devel
