2017-11-03 11:34 GMT+01:00 Julien Cristau <jcris...@debian.org>:
> On 11/03/2017 11:25 AM, Julien Cristau wrote:
>> Package: appstream
>> Severity: grave
>> X-Debbugs-Cc: ftpmas...@ftp-master.debian.org
>>
>> [filing this against the appstream package as a substitute for
>> appstream.debian.org, feel free to reassign if there's a better place]
Thanks! The appstream-generator package is a much better place for
this bug. I'll also reduce the severity of this bug ti important, so
it doesn't impact the package in testing, as this issue appears only
in Debian so far.
>> Hi,
>>
>> there's some archive breakage because
>> dists/sid/non-free/dep11/Components-amd64.yml.xz and
>> dists/sid/non-free/dep11/Components-amd64.yml.gz are out of sync. It
>> looks like appstream-generator silently failed to write the xz version
>> and left a mess in
>> /srv/appstream.debian.org/workspace/public/data/sid/non-free/ on mekeel.
>> This should be made more robust, and any failure should mean nothing
>> gets published rather than risking out of sync metadata.
Agreed. I guess I could rsync the files locally to a separate location
for the archive to fetch them from (this might require a small patch
to dak though).
> from the log (reordered for clarity):
>
> 2017-11-02 21:20:13 - INFO: Writing metadata for sid/non-free [amd64]
> object.Exception@../src/asgen/zarchive.d(448): Unable to open file
> '/srv/appstream.debian.org/workspace/public/data/sid/non-free/Components-amd64.yml.xz.new':
> lzma compression failed: lzma_code() call returned status 11
I've seen this one once before, and I have no idea how this issue
could be triggered. Some other people seem to get status 11 when
writing to a directory they don't have permission to, or when the disk
is full (that's not the case here).
A thing that would definitely help to rule out some issues is to apply
the patch I've sent to the DSA to change the metapackage for mekeel,
so I can finally drop the remaining vendored libraries and start
compiling the binary on the server.
I've also adjusted the generator to be slightly less greedy when
caching data and I've run it multiple times without hitting any issue
- even the memory usage was quite low.
Additionally, I've split the data extraction phase from a per-suite
run to a per-component run. This is less efficient (the generator will
run longer, because it has to reinitialize its icon-lookup table and
cache three times instead of once and has to update some statistical
data multiple times), but ensures that we finalize the LMDB cache
faster and keep a bit less stuff in memory, so the memory usage should
be reduced even more.
This feature was originally created to make asgen testing easier, but
it makes sense to use it at Debian.
This will help in general, but I have no idea on whether this issue is
actually the result of any out-of-memory situation - I think that's
not the case, but I also have no other idea what the problem could be
here. I have only seen this issue two times and I have never been able
to reproduce it.
There is also a (really small) chance that this is a bug in libarchive.
Anyway, I have scheduled a full reprocessing of archive metadata,
let's see what happens (those full data updates are usually the runs
that take huge amounts of memory and are more likely to fail (although
that never happens locally)).
Cheers,
Matthias
