Re: Adding Passim as a Fedora 40 feature?

[Jonathan Dieter]

On Wed, 2023-09-06 at 22:33 +0100, Jonathan Dieter wrote:
> On Fri, 2023-08-25 at 12:42 +0100, Richard Hughes wrote:
> > The tl;dr: is I want to add a mDNS server that reshares the public
> > firmware update metadata from the LVFS on your LAN. The idea is that
> > rather than 25 users in an office downloading the same ~2MB file from
> > the CDN every day, the first downloads from the CDN and the other 24
> > download from the first machine. All machines still download the
> > [tiny] jcat file from the CDN still so we know the SHA256 to search
> > for and verify.
> 
> I realize I'm late to the conversation, but what about compressing the
> metadata using zchunk, like we do for the DNF metadata?  Assuming we
> keep a cache of the file locally and that changes (as a percentage of
> the whole file) are minimal, this allows you to download only the
> differences.  The only requirement is that the CDN supports HTTP range
> requests.
> 
And, of course, after posting, I realize that I'd missed a chunk of the
thread where you explained that you're not a fan of deltas.  FWIW,
zchunk doesn't do static deltas, so the only file you need to worry
about on the server/CDN is the latest one.

If this is something you'd be interested in, I'd be happy to help get
it working.  If not, I'm happy to fade back into the background. :)

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Adding Passim as a Fedora 40 feature?

[Jonathan Dieter]

On Fri, 2023-08-25 at 12:42 +0100, Richard Hughes wrote:
> The tl;dr: is I want to add a mDNS server that reshares the public
> firmware update metadata from the LVFS on your LAN. The idea is that
> rather than 25 users in an office downloading the same ~2MB file from
> the CDN every day, the first downloads from the CDN and the other 24
> download from the first machine. All machines still download the
> [tiny] jcat file from the CDN still so we know the SHA256 to search
> for and verify.

I realize I'm late to the conversation, but what about compressing the
metadata using zchunk, like we do for the DNF metadata?  Assuming we
keep a cache of the file locally and that changes (as a percentage of
the whole file) are minimal, this allows you to download only the
differences.  The only requirement is that the CDN supports HTTP range
requests.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Orphaning lizardfs

[Jonathan Dieter]

That's wonderful news!  I've moved roles since adding it to Fedora and
no longer use it in my day job, so I'm more than happy to let you have
it!

Thanks,

Jonathan

On Sat, 2023-01-28 at 21:25 -0500, JT wrote:
> I can pick this package up if you're stepping back.  The lizards devs
> are planning on a new release sometime this year, but there's still a
> few things they're trying to finish up first before releasing v3.13.
> Last I heard they wanted it to be out before summer.
> JT
> 
> On January 28, 2023 10:14:14 AM Jonathan Dieter 
> wrote:
> 
> > I've just orphaned lizardfs.  Lizardfs is a clustered network
> > filesystem that has very efficient small file / metadata
> > performance,
> > but hasn't seen any upstream point releases since the end of 2017
> > and
> > now FTBFS in the latest mass rebuild.
> > 
> > Jonathan
> > ___
> > devel mailing list -- devel@lists.fedoraproject.org
> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-
> > US/project/code-of-conduct/
> > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/devel@lists.fedorapro
> > ject.org
> > Do not reply to spam, report it: https://pagure.io/fedora-
> > infrastructure/new_issue
> > 
> 

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Orphaning lizardfs

[Jonathan Dieter]

I've just orphaned lizardfs.  Lizardfs is a clustered network
filesystem that has very efficient small file / metadata performance,
but hasn't seen any upstream point releases since the end of 2017 and
now FTBFS in the latest mass rebuild.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]

[Jonathan Dieter]

On Thu, 2022-12-01 at 00:41 +, Daniel Alley wrote:
> 
> * zchunk and deltarpm both reimplement / "bundle" multiple different
> hashing algorithms

zchunk does have bundled versions of various hashing algorithms, but,
if it's compiled against OpenSSL (as it is in Fedora), it uses the
OpenSSL hashing algorithms rather than the bundled versions.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Intent to retire: novacom-client, novacom-server

[Jonathan Dieter]

On Tue, 2022-08-16 at 22:28 -0500, Maxwell G via devel wrote:
> On Tuesday, August 16, 2022 Jonathan Dieter wrote:
> > So, unless I hear from someone who wants it within the next week and
> > has a plan on how to fix the current FTBFS bug[2], on August 23, I will
> > retire novacom-client[3] and novacom-server[4].
> 
> I would suggest just orphaning the packages. This way, anyone can pick up the 
> packages themself through the src.fp.o web interface. The packages will get 
> automatically retired in six weeks if nobody claims them.
> 

Ok, that's done now.  I've orphaned both novacom-client and novacom-
server.  If someone wants them, they're all yours.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Intent to retire: novacom-client, novacom-server

[Jonathan Dieter]

In the deep mists of time, I managed to get a HP Touchpad in the Great
Fire Sale of 2011[1] and got to experiment with the Glorious System of
Operation that was webOS.  Next, naturally, was packaging up novacom,
the webOS USB management tool (think the WebOS equivalent of adb), for
Fedora.  Despite selling my Touchpad a couple of years later, I've
continued to maintain novacom, mainly due to the fact that there was
very little work involved.

Unfortunately, this state of affairs has ended.  Libusb, one of
novacom's dependencies has been retired, and it's not clear to me that
it's worthwhile (or even possible) to keep novacom alive.

So, unless I hear from someone who wants it within the next week and
has a plan on how to fix the current FTBFS bug[2], on August 23, I will
retire novacom-client[3] and novacom-server[4].

Jonathan

[1] https://tedium.co/2020/03/31/hp-touchpad-history/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2113552
[3] https://src.fedoraproject.org/rpms/novacom-client
[4] https://src.fedoraproject.org/rpms/novacom-server
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Orphaning deltarpm

[Jonathan Dieter]

Hi everyone,

I'm orphaning deltarpm because, as it's currently used in Fedora, it's
not very effective, bugs keep getting opened against it because it's
not working as well as it should (mostly an infra issue as opposed to a
problem with the tool itself), and I no longer have the time or
motivation to deal with these tickets when it seems like there's not
much benefit.

The main problem with deltarpms is that we don't keep them from compose
to compose, so we're only getting the previous day's deltarpms when we
download updates.  This means that the only way you'll be able to truly
take advantage of deltarpms is if you get updates after every compose.

For more details, see:
https://pagure.io/releng/issue/7215

Bugs that need to be looked into:

https://bugzilla.redhat.com/show_bug.cgi?id=1873876 - This is mostly
two different problems: 1) when the zstd algorithm changes, deltarpms
cannot be rebuilt to the exact same RPM, and 2) deltarpms for the
kernel package seem to be impossible to rebuild to the exact same RPM
due to the compression format used on the kernel RPMs

https://bugzilla.redhat.com/show_bug.cgi?id=2058477 - No idea why 100+
MB of packages failed to build correctly (no logs in the ticket), but
it's probably related to (1) above.

A brief history of how deltarpms came to be added to Fedora:
https://www.jdieter.net/posts/2013/05/31/eulogy-for-yum-presto/

Finally, I want to say a huge thank you to Michael Schroeder, who wrote
deltarpm.  The tool itself still works beautifully well, and the issues
we're seeing are all infrastructure related.  I also want to say thank
you to everyone who worked on getting deltarpms into Fedora in the
first place and who have helped keep them running this long.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Failed OpenQA tests for zchunk update - how to troubleshoot?

[Jonathan Dieter]

On Sun, 2022-02-20 at 15:37 -0800, Adam Williamson wrote:
> On Sun, 2022-02-20 at 20:26 +0000, Jonathan Dieter wrote:
> > I've just pushed zchunk-1.2.0 to all active Fedora branches, and
> > it's
> > passed the (admittedly non-comprehensive) zchunk test suite, but
> > I'm
> > seeing 2 OpenQA failed tests in Bodhi:
> > 
> > https://bodhi.fedoraproject.org/updates/FEDORA-2022-e4bcaeea7a
> > https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3da9e6213
> > 
> > Looking through
> > https://openqa.fedoraproject.org/tests/1138503#step/_do_install_and_reboot/5
> > it's not clear where the problem is in the screenshots that show as
> > failed, especially since zchunk has nothing to do with the UI.
> > 
> > Is this some unrelated OpenQA error, or am I missing something?
> 
> You're missing that I'm an idiot and JSON is awful. Sorry!
> 
> https://pagure.io/fedora-qa/os-autoinst-distri-fedora/c/9bc039b26de4779f74d5da146889779b2a55b111?branch=master
> 
> Should be all good in an hour or so, after I hit a bunch of restart
> buttons.

Thanks so much for looking into this!  That fixed the F34 build, but
I'm still seeing a failed test in F35:
https://openqa.fedoraproject.org/tests/1138994

Is there some way I can restart it, or does that require special
permission?

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Failed OpenQA tests for zchunk update - how to troubleshoot?

[Jonathan Dieter]

I've just pushed zchunk-1.2.0 to all active Fedora branches, and it's
passed the (admittedly non-comprehensive) zchunk test suite, but I'm
seeing 2 OpenQA failed tests in Bodhi:

https://bodhi.fedoraproject.org/updates/FEDORA-2022-e4bcaeea7a
https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3da9e6213

Looking through
https://openqa.fedoraproject.org/tests/1138503#step/_do_install_and_reboot/5
it's not clear where the problem is in the screenshots that show as
failed, especially since zchunk has nothing to do with the UI.

Is this some unrelated OpenQA error, or am I missing something?

Thanks for any insight,

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Starting raid-check.timer renders system unusable

[Jonathan Dieter]

On Sun, 2021-03-21 at 17:52 +, Jonathan Dieter wrote:
> I'm really hoping that I'm missing something obvious here, but I fear
> that a good chunk of our Fedora systems will be unbootable if they're
> rebooted without disabling raid-check.timer.

Just to follow up on this, it appears that the problem is limited to
systems in the Europe/Dublin time zone.  Not good, but not the disaster
I was fearing.  Never been so glad to miss something obvious. ;)

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Starting raid-check.timer renders system unusable

[Jonathan Dieter]

On Sun, 2021-03-21 at 17:52 +, Jonathan Dieter wrote:
> There is a workaround: disabling raid-check.timer, but, if you can't
> boot due to this bug, you have to boot into single-user mode (which
> requires a root password to have been set).

As Tom Hughes just pointed out to me, if you're stuck with an
unbootable system, you can add systemd.mask=raid-check.timer to the
kernel command line.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Starting raid-check.timer renders system unusable

[Jonathan Dieter]

Hey everyone,

For reference, a bug report has been filed at
https://bugzilla.redhat.com/show_bug.cgi?id=1941335

I just wanted to give a heads up that I came across a bug today that
renders Fedora 33 systems unbootable, even after a clean install.  If
systemd starts raid-check.timer, it gets stuck in what looks like a
busy loop, is unable to start new services, systemctl stops responding
to commands, and we end up with a lot of zombie processes.

The problem is that raid-check.timer (a part of mdadm) is part of the
default boot process in Fedora Workstation, and the bug also exists on
the version of systemd installed with F33 GA.

I've tested on four different systems, each running F33 installed in
different ways (some upgrades, some fresh F33 installs), and on each of
them, starting raid-check.timer makes the system unusable.

There is a workaround: disabling raid-check.timer, but, if you can't
boot due to this bug, you have to boot into single-user mode (which
requires a root password to have been set).

Manually experimenting with dates seems to indicate that this bug is
triggered if the current date is after March 3rd, 2021 at 1:00AM, which
is why we haven't seen this bug before today.

I'm really hoping that I'm missing something obvious here, but I fear
that a good chunk of our Fedora systems will be unbootable if they're
rebooted without disabling raid-check.timer.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Delta RPMs in Fedora 34

[Jonathan Dieter]

On Tue, 2021-01-05 at 08:49 -0500, Neal Gompa wrote:
> To be blunt, I would have never done Zchunk metadata if it was going
> to be used as a tool to kill DeltaRPMs. I firmly believe we need both
> to have a comprehensive offering that accommodates the needs of
> Fedora users across the world.

Hey Neal, I'm not sure where you're going with that first sentence, but
I think it's pretty obvious that zchunk and deltarpms solve different
problems and, following this thread, I don't think anyone has suggested
that we should kill deltarpms *because* we have zchunk metadata.

When we first brought deltarpms into Fedora, a savings of 60-90% when
doing updates was normal.  Now that we're losing the deltarpms after
each push (as we have been for the last three years), the savings is
significantly lower (I normally see less than 10%) and that makes it
hard to be motivated to fix the bugs that inevitably arise.

It sounds like there might be a plan to keep deltarpms beyond a single
push, and, if that happens, I will be more than happy to keep on
dealing with deltarpm bugs. :)

Thanks,
Jonathan


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)

[Jonathan Dieter]

On Sat, 2021-01-02 at 18:12 +, Jonathan Dieter wrote:
> FWIW, I also think it's time for drpms to go.  Aside from any potential
> issues with the proposed change, they haven't been useful in Fedora for
> three years, (see https://pagure.io/releng/issue/7215), and nobody's
> been able to put in the time to fix it yet.  If that changed and
> someone was willing to step up and commit to fixing this, I'd feel very
> differently.
> 
> In addition, drpms aren't even working at the moment.  Something has
> changed during the last week or so that's broken them (see
> https://bugzilla.redhat.com/show_bug.cgi?id=1911828).  I'll take a
> look, but, being honest, there's not much motivation to investigate
> this when drpms are of such marginal use in Fedora at the moment.
> 
> Jonathan

Apologies for the odd quoting in the previous email; Evolution decided
that what you see isn't what you get. :)  I've trimmed out everything
but my response here.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)

[Jonathan Dieter]

On Sat, 2021-01-02 at 13:42 +, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Dec 30, 2020 at 10:10:27AM -0800, Kevin Fenzi wrote:
> 
> This is most likely because we are only making drpms against the most
> recent updates. So, we are making very few drpms and only against
> things
> that recently updated. 

So... people who actually care about the total download are likely not
to update all the time, which also means that drpms will not work for
them.

> For example:
> https://dl.fedoraproject.org/pub/fedora/linux/updates/33/Everything/x86_64/drpms/
> (126 drpms for all of f33 updates). 
> 
So... that means that drpms wouldn't even make a difference for people
who update often.

...and the proposed Change would require additional contortions to
allow
drpms to work. It sounds like drpms are not worth the trouble anymore.
The effort to make them work properly would be large. I think the
crucial bit is that we have more packages and updates than ever, and
at the same time more people update at custom schedules, so any
reasonable
subset of drpms will cover a shrinking subset of upgrades.

Zbyszek

> > Maybe the time has come to just disable DRPM entirely for F34.
> 
> We could. Or try and make them more usefull again. 


FWIW, I also think it's time for drpms to go.  Aside from any potential
issues with the proposed change, they haven't been useful in Fedora for
three years, (see https://pagure.io/releng/issue/7215), and nobody's
been able to put in the time to fix it yet.  If that changed and
someone was willing to step up and commit to fixing this, I'd feel very
differently.

In addition, drpms aren't even working at the moment.  Something has
changed during the last week or so that's broken them (see
https://bugzilla.redhat.com/show_bug.cgi?id=1911828).  I'll take a
look, but, being honest, there's not much motivation to investigate
this when drpms are of such marginal use in Fedora at the moment.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Updated zdicts for F33 zchunk metadata

[Jonathan Dieter]

Thanks Dusty for the pointers on the Freeze Exception process.  I've
created https://bugzilla.redhat.com/show_bug.cgi?id=1886581 blocked
the Final Freeze Exception bug.

Jonathan

On Thu, Oct 8, 2020 at 8:32 PM Jonathan Dieter  wrote:
>
> I'm afraid I'm late to the party on updating the zstd dictionaries
> used by zchunk for F33.  I've just built fedora-repo-zdicts-2010.1,
> which now includes the F33 dictionaries, but we're obviously past
> final freeze.
>
> I would appreciate karma on the following update so it can go in as a
> zero-day update:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9a5b96259
>
> Additionally, if there's any way we can get the GA metadata generated
> on a system with this update installed, the zchunk metadata will be
> significantly smaller.  Just to be clear, it won't cause any breakage
> if we can't, but it would be nice to get the 20-30% reduction in
> zchunk metadata size.
>
> Thanks, and apologies for any inconvenience,
>
> Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Updated zdicts for F33 zchunk metadata

[Jonathan Dieter]

I'm afraid I'm late to the party on updating the zstd dictionaries
used by zchunk for F33.  I've just built fedora-repo-zdicts-2010.1,
which now includes the F33 dictionaries, but we're obviously past
final freeze.

I would appreciate karma on the following update so it can go in as a
zero-day update:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9a5b96259

Additionally, if there's any way we can get the GA metadata generated
on a system with this update installed, the zchunk metadata will be
significantly smaller.  Just to be clear, it won't cause any breakage
if we can't, but it would be nice to get the 20-30% reduction in
zchunk metadata size.

Thanks, and apologies for any inconvenience,

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Jonathan Dieter]

On Wed, 2019-05-29 at 18:05 -0400, Neal Gompa wrote:
> On Wed, May 29, 2019 at 5:53 PM Josh Boyer  wrote:
> > 
> > If we did this, wouldn't it make it very difficult to use tools like
> > mock on RHEL / CentOS 7 to build for Fedora 3x?  Or does RHEL 7 RPM
> > support zstd?
> > 
> 
> We're pretty much screwed here. Also, since RHEL 8's rpm package does
> not have zstd support compiled in, it too cannot handle the RPMs.
> 
> Cf. https://git.centos.org/rpms/rpm/blob/c8/f/SPECS/rpm.spec#_17-18

I just wanted to point out my post[1] in November where I suggested
using zchunk as the compression format for rpm.  IIRC, the main concern
with that proposal was compatibility with RHEL.

The one main advantage using zchunk would have over zstd would be the
ability to completely eliminated drpms, but, as mentioned in that
thread, it would require some changes to the RPM format.

Jonathan

 1: 
https://lists.fedorahosted.org/archives/list/devel@lists.fedoraproject.org/thread/YHKXMJHZW3O6EWA2WYMFWOC22KTVTPLB/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Jonathan Dieter]

On Wed, 2019-05-29 at 18:32 -0400, James Cassell wrote:
> 
> Would this help with drpms similar to how it helps with faster yum
> repo metadata downloads? My biggest problem with drpms is the slow
> rebuild speed which is usually slower than my download bandwidth.  It
> would be a big win if zstd helps here.

Unfortunately not.  The drpm rebuild process involves recompressing the
rpm, so we'd be affected by the compression speed, not the
decompression speed.  With zstd compression level > 15, the drpm
rebuild speed would actually slow down (possibly quite significantly).

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Jonathan Dieter]

On Wed, 2019-05-29 at 20:15 -0600, Chris Murphy wrote:
> 'dnf info deltarpm' says
> URL  : http://gitorious.org/deltarpm/deltarpm
> which has an expired certificate, but pushing passed that it says
> current version 3.6 is 5 years old. Is this really maintained or
> updatabled?

Upstream has changed to 
https://github.com/rpm-software-management/deltarpm.  The code is still
maintained, but there's not much active development.  I can't speak for
the upstream maintainer, but I would guess that a PR that adds zstd
support would probably be welcomed.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Can we use SCLs for building for EPEL 6?

[Jonathan Dieter]

On Sun, 2019-04-14 at 16:01 -0400, Stephen John Smoogen wrote:
> On Sat, 13 Apr 2019 at 21:06, Todd Zullinger  wrote:
> > Neal Gompa wrote:
> > > If devtoolset is available for EPEL6 (which I think it is?)
> > 
> > I don't believe devtoolset was enabled for el6 in koji.
> > When it was added to the mock configs for el6/el7, the
> > consensus on the epel list was that it would be added to el6
> > if there was sufficient demand.  I've only seen it come up
> > once (or maybe twice) since then on the epel list.
> > 
> > I'm not familiar enough with the koji commands to confirm
> > it.  I can see that rhel7-server-rhscl-7 is listed in the
> > external repos, but I don't see a similar rhel6 SCL.
> > 
> > Apologies if I simply missed an announcement on the epel
> > lists and am passing on outdated data.
> > 
> 
> I believe Todd is correct. At the time there was the package needing
> SCL's was chromium and the owner had no interest for making the
> package in EL6. If zchunk needs it, we can put it in.

That's ok.  I've gone with Kevin Kofler's suggestion and just fixed the
build to work with the old version of GCC.

Thanks, all, for the help!

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Can we use SCLs for building for EPEL 6?

[Jonathan Dieter]

On Sat, 2019-04-13 at 13:11 -0700, John Reiser wrote:
> >  Unfortunately, the gcc in EL6 is too old to build zchunk
> 
> In what specific way(s)?  Can the complaints from gcc [which version?],
> or other tools in the toolchain, be listed here?
> Other developers may have faced the same or similar problems,
> and may have tools to help.

Sorry, I should have shared that the first time around.

The version of gcc that comes with EL6 is 4.4.7.

When building zchunk, I get a number of messages that look like:
$ ninja-build 
[1/178] Compiling C object 'src/lib/zck@sha/comp_zstd_zstd.c.o'.
FAILED: src/lib/zck@sha/comp_zstd_zstd.c.o 
cc  -Isrc/lib/zck@sha -Isrc/lib -I../src/lib -Iinclude -I../include -pipe 
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -O0 -g -DZCHUNK_ZSTD 
-DZCHUNK_OPENSSL -fvisibility=hidden -fPIC -MMD -MQ 
'src/lib/zck@sha/comp_zstd_zstd.c.o' -MF 'src/lib/zck@sha/comp_zstd_zstd.c.o.d' 
-o 'src/lib/zck@sha/comp_zstd_zstd.c.o' -c ../src/lib/comp/zstd/zstd.c
In file included from ../src/lib/comp/zstd/zstd.c:34:
../src/lib/zck_private.h:92: error: redefinition of typedef 'zckCtx'
include/zck.h:49: note: previous declaration of 'zckCtx' was here
../src/lib/zck_private.h:106: error: redefinition of typedef 'zck_log_type'
include/zck.h:47: note: previous declaration of 'zck_log_type' was here
../src/lib/zck_private.h:117: error: redefinition of typedef 'zckHash'
include/zck.h:50: note: previous declaration of 'zckHash' was here
../src/lib/zck_private.h:150: error: redefinition of typedef 'zckDL'
include/zck.h:54: note: previous declaration of 'zckDL' was here
../src/lib/zck_private.h:165: error: redefinition of typedef 'zckChunk'
include/zck.h:51: note: previous declaration of 'zckChunk' was here
../src/lib/zck_private.h:177: error: redefinition of typedef 'zckIndex'
include/zck.h:52: note: previous declaration of 'zckIndex' was here
../src/lib/zck_private.h:193: error: redefinition of typedef 'zckRange'
include/zck.h:53: note: previous declaration of 'zckRange' was here
../src/lib/zck_private.h:224: error: redefinition of typedef 'zckComp'
../src/lib/zck_private.h:91: note: previous declaration of 'zckComp' was here
../src/lib/zck_private.h:298: error: redefinition of typedef 'zckCtx'
../src/lib/zck_private.h:92: note: previous declaration of 'zckCtx' was here

For reference, you can find zck.h.in (which gets processed into zck.h
with the version added) at:
https://github.com/zchunk/zchunk/blob/master/include/zck.h.in

and zck_private.h at:
https://github.com/zchunk/zchunk/blob/master/src/lib/zck_private.h

As far as I can see, gcc-4.7 doesn't like that I'm typedefing the same
struct to the same type twice.  Later versions don't see it as a
problem at all.

(Just to be clear, this still happens if I change zck_private.h to say:
typedef struct zckCtx zckCtx;)

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Can we use SCLs for building for EPEL 6?

[Jonathan Dieter]

So, the background is that I'd like to build zchunk for EPEL 6 (it's
already built for EPEL 7).  Unfortunately, the gcc in EL6 is too old to
build zchunk, so I'd prefer to use a newer version from an SCL, rather
than rewrite zchunk to be compatible with an ancient version of gcc.

I noticed that SCLs are available for EPEL 7 (note the final repository
in the list at https://koji.fedoraproject.org/koji/taginfo?tagID=259),
but not for EPEL 6 (see 
https://koji.fedoraproject.org/koji/taginfo?tagID=140).

So, is there any way we can use SCLs for building on EPEL 6, or should
I stop tilting at windmills and just say EL6 is too old for zchunk?

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

How to fix dnf segmentation fault (zchunk metadata is re-enabled)

[Jonathan Dieter]

FESCo has given us the go-ahead to turn zchunk metadata on again[1] for
the F30 fedora repository after the librepo segmentation fault bug[2]
was fixed.  An updated librepo[3] was built a week ago and was pushed
to stable five days ago, so most beta users should have the new
version.

If you're using F30 and have librepo-1.9.6-1.fc30, please update to
librepo-1.9.6-2.fc30 as soon as possible!  If you don't, you will once
again get a segmentation fault when running dnf.

If you find yourself getting a segmentation fault when running dnf on a
system with librepo-1.9.6-1.fc30, you can temporarily work around it by
doing the following:

1. Edit /etc/dnf/dnf.conf and add the following line:

   zchunk=False

2. Run:
   dnf update librepo

   Verify that you got librepo-1.9.6-2.fc30 or later in the update

3. Remove the 'zchunk=False' line you added to /etc/dnf/dnf.conf

4. Run:
   dnf update

At this point, dnf should download the repository metadata without any
segmentation faults, and any further repository metadata downloads
should be significantly smaller.

Jonathan

 1. https://pagure.io/fesco/issue/2116
 2. 
https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org/thread/RHZ6V6MFVFAIJ6OEXZ5VL7BPPLHJE4GF/
 3. https://bodhi.fedoraproject.org/updates/FEDORA-2019-07c3e09858
___
devel-announce mailing list -- devel-annou...@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

How to fix dnf segmentation fault (zchunk metadata is re-enabled)

[Jonathan Dieter]

FESCo has given us the go-ahead to turn zchunk metadata on again[1] for
the F30 fedora repository after the librepo segmentation fault bug[2]
was fixed.  An updated librepo[3] was built a week ago and was pushed
to stable five days ago, so most beta users should have the new
version.

If you're using F30 and have librepo-1.9.6-1.fc30, please update to
librepo-1.9.6-2.fc30 as soon as possible!  If you don't, you will once
again get a segmentation fault when running dnf.

If you find yourself getting a segmentation fault when running dnf on a
system with librepo-1.9.6-1.fc30, you can temporarily work around it by
doing the following:

1. Edit /etc/dnf/dnf.conf and add the following line:

   zchunk=False

2. Run:
   dnf update librepo

   Verify that you got librepo-1.9.6-2.fc30 or later in the update

3. Remove the 'zchunk=False' line you added to /etc/dnf/dnf.conf

4. Run:
   dnf update

At this point, dnf should download the repository metadata without any
segmentation faults, and any further repository metadata downloads
should be significantly smaller.

Jonathan

 1. https://pagure.io/fesco/issue/2116
 2. 
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/RHZ6V6MFVFAIJ6OEXZ5VL7BPPLHJE4GF/
 3. https://bodhi.fedoraproject.org/updates/FEDORA-2019-07c3e09858
___
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org

PSA: workaround for segfault when running dnf update in F30

[Jonathan Dieter]

librepo-1.9.6 has a major bug that will cause a segfault when a
repository has zchunk metadata.  To temporarily work around the
problem, set zchunk=False in /etc/dnf/dnf.conf or wait until the next
updates push comes out


About a week ago, we disabled zchunk metadata in the main F30
repository because of arm-specific compose problems[1].  The problems
were fixed and we enabled zchunk metadata in F30 updates-testing
yesterday, but, in the interval when there were no zchunk-enabled
repositories, librepo-1.9.6 was released, and it has a major bug where
it segfaults whenever dnf update is run on a repository with zchunk
metadata.

We've disabled zchunk metadata for F30 updates-testing and doing a new
push now.

I have submitted a fix[2] for the bug, but for now any F30 users need
to do one of the following:
 * Wait until the next updates push is out.  It won't have zchunk
   metadata, so updates will work normally again.
 * Set zchunk=False in /etc/dnf/dnf.conf.  This will force dnf to fall
   back to non-zchunk metadata, bypassing the bug.

Many apologies for the inconvenience.

Jonathan

[1] https://pagure.io/dusty/failed-composes/issue/1703
[2] https://github.com/rpm-software-management/librepo/pull/148

___
devel-announce mailing list -- devel-annou...@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

PSA: workaround for segfault when running dnf update in F30

[Jonathan Dieter]

librepo-1.9.6 has a major bug that will cause a segfault when a
repository has zchunk metadata.  To temporarily work around the
problem, set zchunk=False in /etc/dnf/dnf.conf or wait until the next
updates push comes out


About a week ago, we disabled zchunk metadata in the main F30
repository because of arm-specific compose problems[1].  The problems
were fixed and we enabled zchunk metadata in F30 updates-testing
yesterday, but, in the interval when there were no zchunk-enabled
repositories, librepo-1.9.6 was released, and it has a major bug where
it segfaults whenever dnf update is run on a repository with zchunk
metadata.

We've disabled zchunk metadata for F30 updates-testing and doing a new
push now.

I have submitted a fix[2] for the bug, but for now any F30 users need
to do one of the following:
 * Wait until the next updates push is out.  It won't have zchunk
   metadata, so updates will work normally again.
 * Set zchunk=False in /etc/dnf/dnf.conf.  This will force dnf to fall
   back to non-zchunk metadata, bypassing the bug.

Many apologies for the inconvenience.

Jonathan

[1] https://pagure.io/dusty/failed-composes/issue/1703
[2] https://github.com/rpm-software-management/librepo/pull/148

___
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org

Re: Call for zchunk repodata testers

[Jonathan Dieter]

On Thu, 2019-01-17 at 00:48 +0100, Björn 'besser82' Esser wrote:
> Am Mittwoch, den 16.01.2019, 22:08 + schrieb Jonathan Dieter:
> > Just a heads up Rawhide has had zchunked metadata for almost three
> > weeks, and I'd greatly appreciate some more testing on the client side
> > before we finish pushing the client changes to Rawhide.
> > 
> > …
> > 
> > If the second day's repository download size for Rawhide is less than
> > 54MB, you'll know everything's working correctly.
> 
> I've just tested updating from yesterday's metadata and it took me about
> 2.9 MBytes (vs. about 61 MBytes without zchunk support) to download.
> 
> Seems to be working as it should.  LGTM  =)

Thanks for the feedback!  That sounds about right for a normal day's
updates.

Jonathan


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Call for zchunk repodata testers

[Jonathan Dieter]

Just a heads up Rawhide has had zchunked metadata for almost three
weeks, and I'd greatly appreciate some more testing on the client side
before we finish pushing the client changes to Rawhide.

If you're running Rawhide, are willing to test, and have backups of
libdnf and librepo, please enable the COPR by running:

dnf copr enable jdieter/dnf-zchunk
dnf update librepo libdnf

At this point all tools using libdnf will default to zchunked metadata
if it's available.

If the second day's repository download size for Rawhide is less than
54MB, you'll know everything's working correctly.

--- Details, feel free to ignore ---

I've been testing it on Rawhide for the last three weeks or so, and the
minimum I've downloaded has been about 1.2MB (updating every day),
while the max today was about 7.9MB (haven't updated for over a week). 
So far, I haven't hit any bugs, but I've only been regularly testing
DNF in the terminal.  I did do initial testing with PackageKit and
microdnf, and they worked fine too.

Currently only primary.xml, filelists.xml and other.xml are zchunked,
but I've just finished extending zchunk support to all metadata types
in createrepo_c, and the changes should be making their way to Rawhide
sometime in the not-to-distant future.

Again, a huge thank you to everyone who's helped with this, and
especially to Daniel Mach for reviewing the large invasive patches
required to make all this work, Neal Gompa for helping push this
change, and Kevin Fenzi for flipping the switch to start generating
zchunked metadata in Rawhide.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Orphaning pykka

[Jonathan Dieter]

On Wed, 2018-12-12 at 06:10 +, Raphael Groner wrote:
> > I've just orphaned pykka (
> > https://admin.fedoraproject.org/pkgdb/package
> > /rpms/pykka/) as I'm no longer using it.
> 
> Hi Jonathan,
> what do you use instead?
> Regards, Raphael

MPD with local music.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Mon, 2018-12-03 at 22:16 +0100, Jan Pokorný wrote:
> On 16/11/18 22:07 +0000, Jonathan Dieter wrote:
> > The core idea behind zchunk is that a file is split into independently
> > compressed chunks and the checksum of each compressed chunk is stored
> > in the zchunk header.  When downloading a new version of the file, you
> > download the zchunk header first, check which chunks you already have,
> > and then download the rest.
> 
> Just one more thought I reliazed in hindsight, there are ways to cut down
> the installed files in RPM ecosystem, currently with a request to omit
> documentation (%doc tagged files, see --nodocs/--excludedocs).
> 
> Indeed, that's a sort of files you can usually omit without hesitation
> in containers/VMs.  Perhaps there are some more bits that are de facto
> optional without losing anything from the functionality.
> 
> So with clever separation, such bits wouldn't even need to be downloaded
> when they will not eventually make it to the disk.  That might make
> things like customizing a base container image tiny bit more swift,
> e.g. in CI/CD context without many connectivity guarantees (up to
> mirrors anyway).  But might not be worth it if the trade-off
> is already predictably suboptimal in other aspects.

I think this is very interesting and definitely feasible (assuming the
core concept of zchunked rpms is reasonable).

On a tangent, I realized something else about metadata generation (and
I think someone else had picked up on it, but it hadn't quite
registered with me).  Currently generating metadata for RPMs involves
reading the full RPM to calculate the checksum.  With zchunk, the
header checksum is stored at the beginning of the file and is all you
need to verify a zchunk file. 

Jonathan


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Thu, 2018-11-22 at 11:31 -0500, Josh Boyer wrote:
> I'm concerned that this will effectively render EL RPM unable to
> handle any Fedora RPMs at all.  That's both a practical concern, as
> many people develop Fedora using EL and vice versa, and also a broader
> ecosystem concern.  I would very much like for all of our
> distributions to be able to more easily operate together, and this
> effectively forks Fedora off into it's own space yet again.

For what very little it's worth, a zchunked rpm is still a valid zchunk
file, and you would be able to easily extract the payload and the
header from a zchunked rpm on an EL system.  In fact, because we're not
planning to change the rpm header format at all, there could easily be
a tool to convert a zchunked rpm into an xz or gzipped rpm (and vice
versa).  Having said that, there's a huge difference between this and
having EL's RPM actually be able to read Fedora's RPMs.

> Have we really looked at the wider scope of what a format change like
> this would do in the context of some of the larger picture things
> we're working on with lifecycle and cross-distro collaboration
> efforts?  I agree this would be better than delta RPMs when looking at
> that *specific* usecase, but improving that (even with compose time
> benefits) by doing a format change seems to be inflicting a very high
> cost for what is an important but relatively small usecase.

This is a really good point and I don't know know the answer.  As per
Neal's suggestion earlier in the thread, I posted this to the rpm-
ecosystem mailing list.  Michael from SUSE brought up some very valid
concerns about how well zchunk would compare with deltarpm in delta
efficiency, but apart from that, it's been pretty quiet there.

If it's already obvious that the cost for this proposal isn't worth the
gain, I do completely understand.  If we're not sure, I'll do a proof-
of-concept that converts standard rpms into zchunked rpms, so we can
compare sizes and deltas, and hopefully get some data points on speed.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Wed, 2018-11-21 at 14:36 +0100, Kamil Paral wrote:
> On Fri, Nov 16, 2018 at 11:13 PM Jonathan Dieter  wrote:
> > For reference, this is in reply to Paul's email about lifecycle
> > objectives, specifically focusing on problem statement #1[1].
> > 
> > 
> > Have rpm use zchunk as its compression format, removing the need for
> > deltarpms, and thus reducing compose time.  This will require changes
> > to both the rpm format and new features in the zchunk format.
> > 
> 
> Hey Jonathan,
> 
> thanks for working on this. The proposed changes sound good to me.
> I'm a bit worried that zchunk is not yet a proven format, so it might
> be a good idea to use it for metadata first, see whether it works as
> expected, and then push it for RPM files. But that's for more
> technical people to judge.
> 
> I have some concrete questions, though:
> 1. I have noticed that especially with large RPMs (firefox, chrome,
> atom, game data like 0ad-data, etc), my PCs are mostly bottlenecked
> by CPU when installing them. And that's with a modern 3.5+GHz CPU.
> That's because RPM decompression runs in a single thread only, and xz
> is just unbelievably slow. I wonder, would zchunk used as an RPM
> compression algorithm improve this substantially? Can it decompress
> in multiple threads and/or does it have much faster decompression
> speeds (and how much)? I don't care about RPM size increase, but I'd
> really like to have them installed fast. (That's of course just my
> personal preference, but this also affects the speed of mock builds
> and such, so I think it's relevant.)

The zstd compression that zchunk uses internally is designed to be
faster than even gzip at decompression.  Currently zchunk is single-
threaded, but, given that each chunk is independent, making it multi-
threaded should be pretty trivial, and is on the todo list.

> 2. In our past QA efforts in Fedora, we had use cases for retrieving
> rpm header data without retrieving the actual content (the payload).
> That was for cases when we needed to check e.g. dependency issues,
> but the rpms were not placed in a repository yet (i.e. no easy access
> to their metadata) and it was slow and wasteful to download the whole
> rpm just to get the header. Will the new zchunk compression still
> make it possible to retrieve just the header without accessing all
> the payload data? (It would be great to make this accessible from
> Python and not just C, but that's a plea I should direct to rpm
> maintainers, I guess).

The zchunk format supports the concept of multiple independent streams
in a single file.  A zchunk rpm would contain two streams, the rpm
header and the rpm payload.  Since downloading a zchunk file is two
steps already (downloading the zchunk header, and then downloading the
required chunks), it should be easy enough to download only the chunks
needed for the rpm header stream.

As for a python API, I would love for zchunk to have that too, but
haven't had the time yet.

I hope that helps.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Wed, 2018-11-21 at 11:31 -0500, Colin Walters wrote:
> After having introduced a new format (OSTree) into the ecosystem here,
> as well as working a lot on the Docker/OCI ecosystem, one thing I want
> to emphasize is:
> 
> A lot of Red Hat's customers don't connect their systems to the Internet,
> they want easy offline mirroring.  OSTree supports that, and it's
> also possible to do with OCI images of course.
> 
> But, a lot organizations use e.g. https://jfrog.com/artifactory/
> which today doesn't support OSTree (it does support RPM and Docker/OCI).
> So any format break for RPM wouldn't be usable until Artifactory gains
> support for it.  And even after that happened you'd have in some
> places a large lag time for it to be deployed.
> 
> In general, any data format break is going to impose a lot higher
> costs than you might imagine.

Thanks for bringing up these points.  You are undoubtedly correct that
there's an unknown cost associated with these changes, but hopefully
the cost will become a little clearer once we have a POC.

> (Also on this topic, I should note that the OSTree data format cleanly
>  fixes a lot of the issues being discussed here; it has deltas, and also
>  doesn't make the mistake of checksumming compressed data,
>  when performing updates only changed files are rewritten, not to mention
>  a whole transactional update system, etc.)

Yep.  I've experimented with OSTree and love the concepts behind it.  I
don't think we're quite ready to ditch the classic rpm systems yet,
though.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Tue, 2018-11-20 at 12:45 +, Michael Schroeder wrote:
> On Mon, Nov 19, 2018 at 08:30:14PM +0000, Jonathan Dieter wrote:
> > Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't
> > require extra CPU usage on the client side as they don't go through the
> > decompress-recompress cycle that deltarpms do.  Re-assembling a zchunk
> > file requires no compression or decompression.
> 
> Btw, we can easily do that for deltarpms as well. We only recompress
> because we want a rpm that is bit-identical to the remote one.
> 
> Having a '-u' option that makes applydeltarpm write a rpm with an
> uncompressed payload and no payload signatures is just a couple of
> lines of code.

But the problem is that you would lose the signatures.  To make this
work, we would need to create signatures of both the compressed and
uncompressed rpm (which wouldn't be a bad idea).  Is there some way we
could (ab)use the current rpm format to make this work, or would it be
a backwards-incompatible change?

Jonathan

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

Hey, thanks for the detailed analysis.  Comments inline.

On Tue, 2018-11-20 at 00:47 +0100, s...@gmx.com wrote:
> Based on work I've done in this area, I'm somewhat sceptical that this
> will work out well.  I spent a decent amount of time comparing various
> approaches to data saving for both compressed data and regular binaries.
> This was done a few months ago, so a few of the algos may have changed a
> little since then, but I doubt the changes will be huge.

Do you have some examples of the data you tested?
> 
> The various things I tested were, in no particular order:
>  * xdelta3 (and a few similar VCDIFF based algos)
>  * zstd 
>  * zchunk
>  * bsdiff
>  * courgette
>  * zucchini
> And a few others which aren't interesting enough to mention here.
> 
> xdelta3 is old now, but was the standard binary delta compression tool
> for a long time.  Of the better performing algos it has reasonably good 
> generation time, and works reasonably well on compressed data (still a 
> good way behind the top of the pile), but falls far behind on binaries. 
> 
> zstd was generally disappointing for both compressed and binary data.  It's
> really a compression format, not a delta generator, and while it performed 
> very well compared to traditional compression formats, it was unable to
> compete with other tools here.
> 
> zchunk had basically the same problems as zstd,  except that the chunking
> overhead made files even larger, and small symbol changes could mean that
> a very large number of chunks needed to be transmitted.
> 
> bsdiff is a larve improvement over xdelta3 in terms of final size for
> binary data.  It is considerably more expensive in terms of memory and cpu
> to compute, but is fast to apply[0]. It works much less well on compressed
> data, and really needs to work on a binary to be at its best.
> 
> courgette is excellent at reducing binary size, and still very good at
> compressed data.  By data size alone it's the best of all of these, but is
> somewhat complex and expensive, particularly in terms of memory.  An
> over view of how it works is [1]
> 
> zucchini is an experimental delta generation tool for chromium.  I can't
> see much that's been published externally about it, but I found it to 
> compress almost as well as courgette but with greatly reduced memory use.
> Code is very much a moving target, but is located at [2].

These seem to be a bit apples-to-oranges.  xdelta3, bsdiff, courgette
and zucchini are designed to generate deltas between two specific
files.  zstd is just a compression format, while zchunk is designed to
download the smallest difference between two arbitrary files.

You are absolutely right about the cost of small symbol changes, one of
the reasons that courgette does some amount of disassembly and that
bsdiff uses what it calls an "add block".  Because of zchunk's design,
there's no way it can benefit unless we implement some kind of
disassembly, and I'm leery of going down that road.

> Overall in my tests zstd/schunk gave massively worse compression compared
> to modern delta algorithms (courgette, zucchini), and the total time to
> download, apply, install was always slowest for zstd based approaches.

I'd love to see your test methodology.  I'm not surprised about the
difference in delta size (which really isn't the same as compression),
but I am a bit surprised by the speed differences you saw, assuming
your data was compressed.

If your data wasn't, do remember that rpms are compressed, so if you're
creating a new delta method, it will have to decompress both the old
and new rpms before generating the delta (as deltarpm does).

zchunk is able to get around this extra step because it is the
compression format.

> So, then, the only time this would work is if the changed-chunk detection
> feature of zchunk actually works effectively for RPMs.  Unfortunately,
> when binaries change (and when RPMs as a whole change) it's not unusual
> for this to manifest as adding/removing significant chunks of data from
> near the beginning of the file, which means that the chunk matching fails
> and you end up with a huge and slow download.

Thats... not how the chunk matching works.  The chunking function uses
the buzhash algorithm to try to break on consistent patterns, no matter
where data is added or removed.  If you remove x bytes from the
beginning of the file and then add y bytes, at most one or two chunks
should be affected.

> If you care only about making the 50th %ile case better, then zchunk is
> probably at least not any worse than what we have now.  However this
> will, I believe, come at the cost of making the 95th %ile case pretty
> unpleasant for end users.
> 
> I think it'd be far better to explore Howard's proposal, for per-file
> delta generation (as opposed to per-RPM), and use modern delta
> generation (probably courgette) to compute the delta.

Maybe I misunderstood Howard's proposal, but I understood him to be
suggesting that rebuilding a full rpm 

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Mon, 2018-11-19 at 16:29 -0500, Simo Sorce wrote:
> On Mon, 2018-11-19 at 21:02 +0000, Jonathan Dieter wrote:
> > On Mon, 2018-11-19 at 15:18 -0500, Simo Sorce wrote:

> > > I do not think you can just trust random metadata somewhere, one of the
> > > points of a rpm reinstall is to fix damaged files for example. It does
> > > no good if you skip those because some file somewhere says they are
> > > "OK". (If I understood your comment about "just downloading changed
> > > chunks).
> > 
> > Yes, this is the crux of the problem.  As I see it, dnf should verify
> > the checksums on the local files before downloading the missing chunks,
> > but that doesn't guarantee that the data won't be changed between the
> > download step and the install step.  RPM would also need to verify the
> > checksums before starting the install phase, and would need to bail out
> > if the checksums had changed.
> > 
> > My biggest concern, though, is what happens if package A needs a
> > specific chunk in /usr/bin/foo and package B changes /usr/bin/foo while
> > being installed.  The chunk was there when the install phase started,
> > but disappeared before package A was actually installed.
> 
> Is this different in a normal install ?
> What if package A installs /usr/bin/foo and then package B overwrites
> it ?
> 
> Or are you concerned about the case where there may be an identical
> chunk in different files ? Are chunks "global" to the host ?

This.  If we stored the checksums in the rpm database, then, yes they
would be global to the host.

> This problem could be addressed by copying all uncompressed chunks in a
> staging area before installing the rpm, failing in a clean way (ie not
> half way through a package install). The penalty is the need for enough
> space to copy the uncompressed files though. more clever things could
> be done with proper filesystem support and snapshotting and copy-on-
> write, but not sure it is worth optimizing for what is normally a
> relatively small scratch area (if you do it one package at a time
> only).

What about just copying any uncompressed chunks required for the
current package or any packages still in the install queue?  That might
reduce the scratch area even further.

> > > 2) what are the chunks sizes ?
> > 
> > The chunk sizes vary because you don't want inserting or removing a few
> > bytes to completely change all the following chunks.  The current
> > default average size is 32KB, but that can be adjusted.
> 
> Is this a compromise between compression performance and granularity ?
> Anything else went into the decision to settle around 32k ?
> Some filesystems seem to gravitate around 64k extents so I am
> wondering.

Yes, this is just a compromise.  The larger the chunk size, the larger
the delta you need to download, but the better the compression.  We
could experiment with this to see if 64k would give us significantly
better compression.

I would also chunk on file borders in the rpm payload, so we don't end
up having a chunk span multiple files.  That would get messy fast when
trying to rebuild from local files.

> > > Finally what signature scheme where you planning to use ? And how do
> > > you deal with the data you want to "exclude" from signing, omit it or
> > > feed in blank "sectors" ?
> > 
> > I was planning to use GPG signatures, and was planning to just omit the
> > data I want excluded.  Having said that, while the format supports
> > signatures, the code hasn't been written and if either of those answers
> > are bad/dangerous, we can change that.
> 
> We use GPG signatures right now, can't be any more dangerous than that
> :-)
> 
> The omission vs blanking has no ill effect, but was not explicitly
> mentioned, it should. Esp around places where the missing data is in
> the middle of a "structure" in your diagrams, or it may be ambiguous
> and lead to incompatible implementations if someone is ever going to
> build another (and if zchunk is going to be adopted in rpm I bet there
> will be some other implementation to do some crazy thing :-)

Yep.  Let me clarify that in the format definition (and add the new
checksum types, I noticed they're missing).

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Mon, 2018-11-19 at 22:18 +0100, Nicolas Mailhot wrote:
> Le lundi 19 novembre 2018 à 20:30 +0000, Jonathan Dieter a écrit :
> > The zchunk advantage over deltarpms is much less CPU usage, while its
> > disadvantages are slightly larger network usage and increased disk
> > usage.
> 
> Unfortunately, that's a bad compromise for most limited clients. A
> limited client can trade time for CPU or network performance, swap for
> memory. What it can absolutely not make more of is storage, both install
> and staging storage. Install storage requirements do not depend on rpm
> tech level, but will generally go up over a system lifetime, adding
> pressure on staging storage.
> 
> So you absolutely need to keep staging storage on par or less than
> existing rpm/dnf if you do not want to obsolete classes of Fedora
> hardware.
> 
> And Google released a huge quantity of cheap storage-deficient hardware
> with its chromebooks. People do install Fedora on those.

The only way we can keep staging storage down (and it would actually be
less than deltarpm/normal rpm) is to use the local chunks at install
time rather than download time.  This comes with its own risks though,
see the other emails in this thread, specifically the ones following
Jan Pokorný's message.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Mon, 2018-11-19 at 21:14 +, Tom Hughes wrote:
> On 19/11/2018 20:30, Jonathan Dieter wrote:
> 
> > On the client:
> > The zchunk advantage over regular rpm is decreased network usage, while
> > its disadvantage is increased disk usage (since the local chunks will
> > be uncompressed).
> > 
> > The zchunk advantage over deltarpms is much less CPU usage, while its
> > disadvantages are slightly larger network usage and increased disk
> > usage.
> > 
> > Note that for most users the increased disk usage is temporary, since
> > rpms are deleted after install.
> 
> If they're deleted after install then surely next time there is an
> update there won't be any local chunks and everything will have to
> be downloaded?
> 
> That's what has been confusing me about this whole thing - as I
> understand it the idea is to only download new chunks and to reuse
> chunks that are unchanged from earlier revisions, but it seems
> that doing that would require keeping a local copy of every
> installed rpm which would be a big change that nobody seems to
> have mentioned.

The idea is to use the locally installed files as the local chunks, the
same way that deltarpm does.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Mon, 2018-11-19 at 15:18 -0500, Simo Sorce wrote:
> On Mon, 2018-11-19 at 19:58 +0000, Jonathan Dieter wrote:

> > That's an interesting thought.  I was picturing using the zchunk
> > library in the dnf download stage to build a local rpm from the
> > verified locally installed files and the downloaded changed chunks,
> > but, if I understand your suggestions correctly, you're saying we
> > could
> > just download the changed chunks and have RPM automatically get the
> > rpm-integrity verified chunks during the *install* stage.
> 
> How do you know which chunks to download w/o having a stored (or
> recomputed) list of existing chunks ?

I thought we should store the chunk checksums of installed files in the
rpm database.  Something like file, offset, length, checksum type,
checksum?

> > The advantage of this method is that you don't need to store the local
> > data twice, but the danger is that the local files get changed
> > elsewhere during the install process.
> > 
> > It's an interesting thought, though, and I wonder if there's a way we
> > could work around that danger?
> 
> I do not think you can just trust random metadata somewhere, one of the
> points of a rpm reinstall is to fix damaged files for example. It does
> no good if you skip those because some file somewhere says they are
> "OK". (If I understood your comment about "just downloading changed
> chunks).

Yes, this is the crux of the problem.  As I see it, dnf should verify
the checksums on the local files before downloading the missing chunks,
but that doesn't guarantee that the data won't be changed between the
download step and the install step.  RPM would also need to verify the
checksums before starting the install phase, and would need to bail out
if the checksums had changed.

My biggest concern, though, is what happens if package A needs a
specific chunk in /usr/bin/foo and package B changes /usr/bin/foo while
being installed.  The chunk was there when the install phase started,
but disappeared before package A was actually installed.

> A couple more questions.
> I skimmed quickly at the format and I have two questions I did not
> immediately see an answer for.
> 1) why are you still supporting SHA-1 in a new format ?

Zchunk cares about two types of checksums, the chunk checksums, used to
determine if two chunks are the same, and the full data checksum (which
currently defaults to SHA-25), used to actually validate the data.

Originally, SHA-1 was supposed to be used *only* for the chunk
checksums, but, somewhere along the way, it was pointed out that using
the first 128 bits of a SHA-512 hash would be faster and more secure,
so the default for the chunk checksums is now SHA-512/128.

The only reason SHA-1 support is still in zchunk is because I don't
want to break backwards compatibility for the (probably five) zchunk
files created before this change.

Having said that, zchunked rpms won't be able to depend on the full
data checksum (because the local chunks will be uncompressed), so we'd
need to use SHA-256 at minimum for the chunk checksums.

> 2) what are the chunks sizes ?

The chunk sizes vary because you don't want inserting or removing a few
bytes to completely change all the following chunks.  The current
default average size is 32KB, but that can be adjusted.

> Sorry if this is already answered somewhere.
> 
> Finally what signature scheme where you planning to use ? And how do
> you deal with the data you want to "exclude" from signing, omit it or
> feed in blank "sectors" ?

I was planning to use GPG signatures, and was planning to just omit the
data I want excluded.  Having said that, while the format supports
signatures, the code hasn't been written and if either of those answers
are bad/dangerous, we can change that.

> Thanks for any answer.
> Simo.

Thank you for looking at this!

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Sun, 2018-11-18 at 13:19 -0500, Stephen John Smoogen wrote:
> On Sun, 18 Nov 2018 at 12:49, Neal Gompa  wrote:
> > On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter  > > wrote:
> > > On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote:
> > > > Jonathan Dieter wrote:
> > > > > My proposal would be to make zchunk the rpm compression
> > > > > format for
> > > > > Fedora.
> > > > 
> > > > Given that:
> > > > 1. zchunk is based on zstd, which is typically less efficient
> > > > in terms of
> > > >compression ratio than xz, depending on settings
> > > >(see, e.g., https://github.com/inikep/lzbench), and
> > > > 2. zchunk can by design only compress chunks individually and
> > > > not benefit
> > > >from the space savings of a solid archive with a global
> > > > dictionary,
> > > > I fear that this is going to significantly increase the size of
> > > > the RPMs,
> > > > which matters:
> > > > * for the initial downloads,
> > > > * for storage (e.g., keepcache=1, local mirrors, etc.), and
> > > > * for the people not using deltas for whatever reason.
> > > > 
> > > > I think zchunk makes a lot of sense for the metadata, but I am
> > > > not convinced
> > > > that it is the right choice for the RPMs themselves.
> > > 
> > > I suspect the first is true, but zchunk does actually allow for a
> > > global (per-file) dictionary that can be used to compress each
> > > chunk.
> > > The difficulty is that the dictionary has to stay the same
> > > between file
> > > versions, or the chunk checksums won't match.  There would have
> > > to be
> > > some thought put into how to generate and store the dictionaries.
> > > 
> > > As for how much bigger a zchunked rpm will be compared to an xz
> > > rpm, at
> > > the moment it's a bit hand-wavy.  Based on zchunked repodata work
> > > I've
> > > done, I think we might be looking at a size that's slightly
> > > smaller
> > > than a gzipped rpm.  I won't know for sure until I put together a
> > > proof-of-concept, but I want to make sure that there aren't any
> > > gaping
> > > holes in the proposal before I do that.
> > > 
> > 
> > I did some work several months ago to evaluate zstd compression for
> > RPMs for Fedora, because of the lower memory and CPU usage for
> > (de)compression. However, the average size increase from xz was
> > pretty
> > large (~20% or more on average, and nothing ever was either the
> > same
> > or smaller), even with heavier compression settings. That might
> > have
> > changed a bit with newer zstd releases that offer some more
> > tunables,
> > but I think it'll remain a tough sell on disk space.
> 
> So there are at least 4 legs here:
> CPU usage (in both uncompression install and deltarpm)
> Memory usage per transaction
> Network amount
> Disk amount
> 
> I expect that the best we are going to get in any 'improvement' is
> going to be 3 out of the 4. The xz compression and delta-rpm has a
> cpu/memory tradeoff for disk and network in comparison to gzip but it
> is mostly acceptable if you have fairly modern desktops. However for
> older hardware or lower power systems that tradeoff may not be good.

Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't
require extra CPU usage on the client side as they don't go through the
decompress-recompress cycle that deltarpms do.  Re-assembling a zchunk
file requires no compression or decompression.

On the client:
The zchunk advantage over regular rpm is decreased network usage, while
its disadvantage is increased disk usage (since the local chunks will
be uncompressed).

The zchunk advantage over deltarpms is much less CPU usage, while its
disadvantages are slightly larger network usage and increased disk
usage.

Note that for most users the increased disk usage is temporary, since
rpms are deleted after install.

In our infrastructure:
The zchunk advantage over deltarpms is that they are created in the
build stage and shouldn't take any longer than building a normal rpm,
while deltarpms take quite a while to build.

The disadvantage is that our current rpms use xz compression which is
more efficient at compressing a whole file than zchunk is, so zchunked
rpms will require more disk space.

Hope that helps clarify things.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Mon, 2018-11-19 at 20:16 +0100, Jan Pokorný wrote:
> On 19/11/18 13:13 +0100, Nicolas Mailhot wrote:
> > Le 2018-11-19 12:28, Martin Kolman a écrit :
> > 
> > > Many people might think RAM would not be an issue in 2018, but in
> > > practice there are
> > > and likely always will be memory constrained installation targets,
> > > such as massive deployments
> > > of "small" VMs or the IoT use cases mentioned above.
> > 
> > Sure, that’s the artificial small vm case
> > 
> > The average old/limited hardware is limited in memory, cpu and storage.
> > Therefore if you have one factor to sacrifice it's cpu time because you can
> > always let the CPU run a little longer, but a limited system won't magically
> > grow more memory or more storage.
> > 
> > Storage would not be such a problem is dnf was smart enough to auto
> > partition big upgrades in lots of small partial upgrades, before downloading
> > gigs of data that do not fit on disk.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1609824
> 
> Also, not familiar with zchunk way of doing things, but couldn't
> rpm-integrity-verified installed files be mapped back to "chunks"
> to further aleviate space concerns for the machine receiving
> updates in some cases?

That's an interesting thought.  I was picturing using the zchunk
library in the dnf download stage to build a local rpm from the
verified locally installed files and the downloaded changed chunks,
but, if I understand your suggestions correctly, you're saying we could
just download the changed chunks and have RPM automatically get the
rpm-integrity verified chunks during the *install* stage.

The advantage of this method is that you don't need to store the local
data twice, but the danger is that the local files get changed
elsewhere during the install process.

It's an interesting thought, though, and I wonder if there's a way we
could work around that danger?

Jonathan


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Sun, 2018-11-18 at 07:02 +, Leigh Scott wrote:
> +1 to anything to rid me of deltarpms, I currently have to disable
> this lame default.

The irony is that getting deltarpms into Fedora was largely my
fault.  ;)  Sorry, Leigh.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Mon, 2018-11-19 at 14:57 +0100, Miroslav Suchý wrote:
> Dne 16. 11. 18 v 23:07 Jonathan Dieter napsal(a):
> >1. Downloading a new release of a zchunked rpm would be larger than
> >   downloading the equivalent deltarpm.  This is offset by the fact
> >   that the client is able to work out which chunks it needs no matter
> >   what the original rpm is, rather than needing a specific original
> >   rpm as deltarpm does.
> 
> Does this means bigger requirements for copies of RPMs too? I mean
> for all our repos mirros, for RH Satellites, Spacewalk, Koji, Copr,
> Retrace...

Yes, and that should actually be item 3 in drawbacks.  Zchunked rpms
will be larger than the current xz-compressed rpms, but the actual size
increase is still unknown.  I think we can shoot for roughly the same
size as a gzip-compressed rpm, but I'm not sure.

Mirror storage *might* actually go down, since we no longer need to
store deltarpms, but anything that only stores rpms will definitely
require more space.

Jonathan

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote:
> Jonathan Dieter wrote:
> > My proposal would be to make zchunk the rpm compression format for
> > Fedora.
> 
> Given that:
> 1. zchunk is based on zstd, which is typically less efficient in terms of
>compression ratio than xz, depending on settings
>(see, e.g., https://github.com/inikep/lzbench), and
> 2. zchunk can by design only compress chunks individually and not benefit
>from the space savings of a solid archive with a global dictionary,
> I fear that this is going to significantly increase the size of the RPMs, 
> which matters:
> * for the initial downloads,
> * for storage (e.g., keepcache=1, local mirrors, etc.), and
> * for the people not using deltas for whatever reason.
> 
> I think zchunk makes a lot of sense for the metadata, but I am not convinced 
> that it is the right choice for the RPMs themselves.

I suspect the first is true, but zchunk does actually allow for a
global (per-file) dictionary that can be used to compress each chunk. 
The difficulty is that the dictionary has to stay the same between file
versions, or the chunk checksums won't match.  There would have to be
some thought put into how to generate and store the dictionaries.

As for how much bigger a zchunked rpm will be compared to an xz rpm, at
the moment it's a bit hand-wavy.  Based on zchunked repodata work I've
done, I think we might be looking at a size that's slightly smaller
than a gzipped rpm.  I won't know for sure until I put together a
proof-of-concept, but I want to make sure that there aren't any gaping
holes in the proposal before I do that.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Sat, 2018-11-17 at 14:43 -0600, Chris Adams wrote:
> Once upon a time, Jonathan Dieter  said:
> > The benefit of zchunked rpms is that, when downloading an updated rpm,
> > you would only need to download the chunks that have changed from
> > what's on your system.
> 
> How well do web servers and caches handle range requests?  I haven't
> really paid attention to range requests in a long time; at one point
> IIRC mirrors would often disable them because of "download accelerators"
> that would open multiple connections to download parts of the same ISO
> in parallel (hogging server resources).

When I did the original POC testing, out of Fedora's 150 mirrors, 3
didn't support range requests at all and 3 supported a limited number
of ranges in a single http request.

Zchunk doesn't open extra connections to the server, but instead
combines as many ranges as the server supports into a single request. 
Currently, if a server doesn't support ranges, zchunk will just
download the full file, but this could be changed to try a different
server.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

On Sat, 2018-11-17 at 14:36 -0500, Neal Gompa wrote:
> On Sat, Nov 17, 2018 at 1:15 PM Jonathan Dieter  wrote:
> > Neal, thanks so much for your thoughts on this.  Responses inline:
> > 
> > On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote:
> > 
> > > If we're really considering changing the RPM file format, then we need
> > > a proper discussion on rpm-maint@ and rpm-ecosystem@ mailing lists on
> > > rpm.org. Can you please start a targeted discussion there?
> > 
> > Sure.
> > 
> > > But addressing the specific concrete suggestion here, there's a few
> > > concerns I have:
> > > 
> > > 1. This is a huge format break, which means that for the first time in
> > > a _very_ long time, it would not be possible to reuse RHEL for Fedora
> > > infrastructure _at all_. That's going to be a difficult problem.
> > > There's a large legacy of systems that won't be able to handle that
> > > new format, and unfortunately, rpm is not parallel installable in the
> > > same manner as something like GCC or Python currently. Making it
> > > parallel installable *is* possible (I've done it, and there have been
> > > other attempts before), but it's not a supported thing. This is
> > > probably the thing that would trigger a major version bump for RPM,
> > > since it's a new archive format.
> > 
> > Agreed, that this would be a massive format change and should therefore
> > be a major version bump for RPM.  New versions of RPM should still be
> > able to read and install old-format rpms, but, as you point out, old
> > versions of RPM won't be able to read or install new-format rpms.
> > Unfortunately, I don't see any way around this.
> > 
> 
> I don't think there's a way around it either. I just hope we do better
> than the last time someone tried to do this...

+1

> > > 2. This also means the _entire_ ecosystem of RPM archive parsers will
> > > break. This is not particularly insurmountable, actually, as the RPM
> > > file format was not particularly well documented, and a new format is
> > > an opportunity to revisit some of those old issues and try to do a
> > > better job this go around. But it's still a challenge to deal with.
> > 
> > Yes, this is going to be quite a bit of work.
> > 
> > > 3. When you refer to the rpm cpio, I assume you're referring to only
> > > the archive payload, right? Typically the payload is what is
> > > compressed, and the headers are not. It sounds like you're proposing
> > > both aspects to be compressed, and compressed differently. If we made
> > > the RPM header an uncompressed zchunk stream and the RPM payload a
> > > zstd-compressed zchunk stream, would we be able to support fetching
> > > header deltas for retrieving extra information on the fly? Say, for
> > > example, attributes like arch color, filecap properties, and so on,
> > > that aren't in the rpm-md data for things like transaction tests
> > > without the whole RPM?
> > 
> > Yes, I'm referring the the archive payload as the cpio.  The zchunk
> > format supports the idea of separate data streams, and I was planning
> > to use that to put the headers in one stream and the archive payload in
> > another.  If the header chunks are first in the zchunk file, then they
> > could be read without needing to read any of the rest of the file.
> > And, yes, we could make the header stream uncompressed if that made it
> > easier to parse.
> > 
> 
> Whether it's compressed or not isn't terribly important, but what is
> important is being able to validate the correctness before beginning
> any processing, including decompression.

Absolutely!  This includes both the rpm header and the rpm archive
data, and that's why we store both the compressed and uncompressed
checksums of the chunks.

> > > 4. I'd actually rather make it easier for the header streams to be
> > > fetched instead of trying to make specific attributes easier in the
> > > header payload. History has shown that any attempt at foresight here
> > > tends to fail miserably, and common attributes are already specified
> > > in the rpm-md primary.xml anyway, so if you're fetching the header to
> > > retrieve an attribute, you *need* to do something weird anyway.
> > 
> > The main purpose of putting separate attributes in the zchunk header is
> > so programs like 'file' can determine some basic information about an
> > rpm without needing to parse the full rpm header.  This data would also
> > be in the rpm header, so programs that read t

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

Neal, thanks so much for your thoughts on this.  Responses inline:

On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote:

> If we're really considering changing the RPM file format, then we need
> a proper discussion on rpm-maint@ and rpm-ecosystem@ mailing lists on
> rpm.org. Can you please start a targeted discussion there?

Sure.

> But addressing the specific concrete suggestion here, there's a few
> concerns I have:
> 
> 1. This is a huge format break, which means that for the first time in
> a _very_ long time, it would not be possible to reuse RHEL for Fedora
> infrastructure _at all_. That's going to be a difficult problem.
> There's a large legacy of systems that won't be able to handle that
> new format, and unfortunately, rpm is not parallel installable in the
> same manner as something like GCC or Python currently. Making it
> parallel installable *is* possible (I've done it, and there have been
> other attempts before), but it's not a supported thing. This is
> probably the thing that would trigger a major version bump for RPM,
> since it's a new archive format.

Agreed, that this would be a massive format change and should therefore
be a major version bump for RPM.  New versions of RPM should still be
able to read and install old-format rpms, but, as you point out, old
versions of RPM won't be able to read or install new-format rpms. 
Unfortunately, I don't see any way around this.

> 2. This also means the _entire_ ecosystem of RPM archive parsers will
> break. This is not particularly insurmountable, actually, as the RPM
> file format was not particularly well documented, and a new format is
> an opportunity to revisit some of those old issues and try to do a
> better job this go around. But it's still a challenge to deal with.

Yes, this is going to be quite a bit of work.

> 3. When you refer to the rpm cpio, I assume you're referring to only
> the archive payload, right? Typically the payload is what is
> compressed, and the headers are not. It sounds like you're proposing
> both aspects to be compressed, and compressed differently. If we made
> the RPM header an uncompressed zchunk stream and the RPM payload a
> zstd-compressed zchunk stream, would we be able to support fetching
> header deltas for retrieving extra information on the fly? Say, for
> example, attributes like arch color, filecap properties, and so on,
> that aren't in the rpm-md data for things like transaction tests
> without the whole RPM?

Yes, I'm referring the the archive payload as the cpio.  The zchunk
format supports the idea of separate data streams, and I was planning
to use that to put the headers in one stream and the archive payload in
another.  If the header chunks are first in the zchunk file, then they
could be read without needing to read any of the rest of the file. 
And, yes, we could make the header stream uncompressed if that made it
easier to parse.

> 4. I'd actually rather make it easier for the header streams to be
> fetched instead of trying to make specific attributes easier in the
> header payload. History has shown that any attempt at foresight here
> tends to fail miserably, and common attributes are already specified
> in the rpm-md primary.xml anyway, so if you're fetching the header to
> retrieve an attribute, you *need* to do something weird anyway.

The main purpose of putting separate attributes in the zchunk header is
so programs like 'file' can determine some basic information about an
rpm without needing to parse the full rpm header.  This data would also
be in the rpm header, so programs that read the rpm header wouldn't
care about the attributes in the zchunk header.

> 5. I'm not exactly sure what you mean by zchunk signing...

The zchunk format supports signing, but just for the zchunk header. 
Because the header contains the checksums for each chunk, this
establishes a chain of trust for verifying the whole file.  Which
brings me to...

> 6. I'm wondering why we can't do a perfect reconstruction of the
> original RPM, given two RPM sources that are both zchunked? We can
> pull it off with repodata, so what's different about RPM that makes
> that not doable?

The problem is that, unlike the repodata, once an rpm is installed, the
package file is deleted and the data is only available on the system in
its uncompressed installed form.  If we're trying to use that data to
rebuild an rpm, we have two options.

   1. Compress the data using the same method that was used to create the
  original rpm.  This is what applydeltarpm does, and is why it's so
  heavy on the CPU.
   2. Store the data uncompressed in the rebuilt rpm.  This isn't feasible
  with deltarpm, but, if we store both compressed hashes and
  uncompressed hashes in the zchunk header, we can do this in zchunk. 
  When running checking the signature, zchunk verifies the header
  against the signature first, and then checks each chunk to see if it
  passes *either* the compressed or uncompressed 

Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Jonathan Dieter]

For reference, this is in reply to Paul's email about lifecycle
objectives, specifically focusing on problem statement #1[1].


Have rpm use zchunk as its compression format, removing the need for
deltarpms, and thus reducing compose time.  This will require changes
to both the rpm format and new features in the zchunk format.


*deltarpm background*
As part of the compose process, deltarpms are generated between each
new rpm and both the GA version of the rpm and the previous version. 
This process is very CPU and memory intensive, especially for large
rpms.

This also means that deltarpms are only useful for an end user if they
are either updating from GA or have been diligent about keeping their
system up-to-date.  If a user is updating a package from N-2 to N,
there will be no deltarpm and the full rpm will be downloaded.

*zchunk background*
As some are aware, I've been working on zchunk[2], a compression format
that's designed for highly efficient deltas, and using it minimize
metadata downloads[3].

The core idea behind zchunk is that a file is split into independently
compressed chunks and the checksum of each compressed chunk is stored
in the zchunk header.  When downloading a new version of the file, you
download the zchunk header first, check which chunks you already have,
and then download the rest.

*Proposal*
My proposal would be to make zchunk the rpm compression format for
Fedora.  This would involve a few additions to the zchunk format[4]
(something the format has been designed to accommodate), and would
require some changes to the rpm file format.

*Benefit*
The benefit of zchunked rpms is that, when downloading an updated rpm,
you would only need to download the chunks that have changed from
what's on your system.

The uncompressed local chunks would be combined with the downloaded
compressed chunks to create a local rpm that will pass signature
verification without needing to recompress the uncompressed local
chunks, making this computationally much faster than rebuilding a
deltarpm, a win for users.

The savings wouldn't be as good as what deltarpm can achieve, but
deltarpms would be redundant and could be removed, completely
eliminating a large step from the compose process.

*Drawbacks*
   1. Downloading a new release of a zchunked rpm would be larger than
  downloading the equivalent deltarpm.  This is offset by the fact
  that the client is able to work out which chunks it needs no matter
  what the original rpm is, rather than needing a specific original
  rpm as deltarpm does.
   2. The rebuilt rpm may not be byte-for-byte identical to the original,
  but will be able to be validated without decompression, as explained
  in the next section

*Changes*
The zchunk format would need to be extended to allow for a zchunked rpm
to contain both the uncompressed chunks that were already on the local
system and the newly downloaded compressed chunks while still passing
signature verification.  This would also require moving signature
verification to zchunk.
 
The rpm file format has to be changed because the zchunk header needs
to be at the beginning of the file in order for the zchunk library
figure out which chunks it needs to download.  My suggestions for
changes to the rpm file format are as follows:

   1. Signing should be moved to the zchunk format as described at the
  beginning of this section
   2. The rpm header should be stored in one stream inside the zchunk
  file.  This allows it to be easily extracted separately from the
  data
   3. The rpm cpio should be stored in a second stream inside the zchunk
  file.
   4. At minimum, an optional zchunk element should be set to identify
  zchunk rpms as rpms rather than regular zchunk files.  If desired,
  optional elements could also be set containing %{name}, %[version},
  %{release}, %{arch} and %{epoch}.  This would allow this information
  to be read easily without needing to extract the rpm header stream.

*Final notes*
I realize this is a massive proposal, zchunk is still very young, and
we're still working on getting the dnf zchunk pull requests reviewed. 
I do think it's feasible and provides an opportunity to eliminate a
pain point from our compose process while still reducing the download
size for our users.

[1]: 
https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements#Challenge_.231:_Faster.2C_more_scalable_composes
[2]: https://github.com/zchunk/zchunk
[3]: https://fedoraproject.org/wiki/Changes/Zchunk_Metadata
[4]: https://github.com/zchunk/zchunk/blob/master/zchunk_format.txt
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 

Re: Package Reviews

[Jonathan Dieter]

On Wed, 2018-10-24 at 19:49 +0200, Robert-André Mauchin wrote:
> Hello,
> 
> I'd like some help for a couple of package reviews:
> 
> Review Request: rclone-browser - Simple cross platform GUI for
> rclone 
> https://bugzilla.redhat.com/show_bug.cgi?id=1642570

I've got this one.  You already reviewed duperemove for me.

Jonathan

> Review Request: strawberry - An audio player and music collection
> organizer
> https://bugzilla.redhat.com/show_bug.cgi?id=1642118
> 
> I'll review anything in exchange but I will be on vacation starting
> Sat 27 to 
> Wed 7.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Cannot find -latomic when building for epel7 aarch64

[Jonathan Dieter]

On Sat, 2018-10-20 at 11:17 -0400, Todd Zullinger wrote:
> Jonathan Dieter wrote:
> > On Fri, 2018-10-19 at 21:37 -0700, Josh Stone wrote:
> > > For aarch64, libatomic comes from the gcc-libraries source package,
> > > which I believe only exists to provide runtime support for devtoolset.
> > > It does not have libatomic.so, only libatomic.so.1 and .so.1.2.0.  You
> > > probably need to use devtoolset gcc to actually build+link it.
> > > (Were those SCLs ever enabled for EPEL?)
> 
> They were enabled in koji for EPEL-7.  (In mock, they are
> enabled for EPEL-6 as well.)
> 
> > Ok, thanks for the explanation.  Unless there's an easy BuildRequires I
> > can add, I think I'll just leave duperemove out of EPEL.
> 
> An example of using devtoolset in a spec file is here:
> 
>   
> http://smoogespace.blogspot.com/2018/03/using-red-hat-developer-toolset-dts-in.html
> 
> I don't know if that will work out easily for your situation
> or not.

That did the job perfectly!  Thanks for the pointer.  Builds have just
finished for EPEL.

Jonathan


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Cannot find -latomic when building for epel7 aarch64

[Jonathan Dieter]

On Fri, 2018-10-19 at 21:37 -0700, Josh Stone wrote:
> On 10/19/18 6:19 PM, Robert-André Mauchin wrote:
> > On samedi 20 octobre 2018 00:31:50 CEST Jonathan Dieter wrote:
> > > I'm trying to build duperemove[1] for epel7[2], and it's building on
> > > all the arches except aarch64.
> > > 
> > > I'm BR'ing libatomic, but the error it gives in build.log[3] for
> > > aarch64 is:
> > > /usr/bin/ld: cannot find -latomic
> > > 
> > > All current Fedora release builds were fine.
> > > 
> > > I'm sure I'm missing something obvious, but does anyone have an idea
> > > what's going on?
> > > 
> > 
> > libatomic is 4.8.5 like the gcc version for other arches.
> > On aarch64, libatomic is 8.2.1 whereas gcc is still 4.8.5.
> > Maybe this causes some issues.
> 
> For aarch64, libatomic comes from the gcc-libraries source package,
> which I believe only exists to provide runtime support for devtoolset.
> It does not have libatomic.so, only libatomic.so.1 and .so.1.2.0.  You
> probably need to use devtoolset gcc to actually build+link it.
> (Were those SCLs ever enabled for EPEL?)

Ok, thanks for the explanation.  Unless there's an easy BuildRequires I
can add, I think I'll just leave duperemove out of EPEL.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Cannot find -latomic when building for epel7 aarch64

[Jonathan Dieter]

I'm trying to build duperemove[1] for epel7[2], and it's building on
all the arches except aarch64.

I'm BR'ing libatomic, but the error it gives in build.log[3] for
aarch64 is:
/usr/bin/ld: cannot find -latomic

All current Fedora release builds were fine.

I'm sure I'm missing something obvious, but does anyone have an idea
what's going on?

 [1] https://github.com/markfasheh/duperemove
 [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=30336065
 [3] https://kojipkgs.fedoraproject.org//work/tasks/6089/30336089/build.log
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

[Jonathan Dieter]

On Mon, 2018-08-06 at 16:36 +, Zbigniew Jędrzejewski-Szmek wrote:
> Hi dnf and libsolv developers,
> 
> this mail is a continuation of an FPC [1] and a FESCo [2] tickets.
> 
> A proposal was made is to disallow packages in Fedora from using file
> deps, and to optimize dnf to not load filelists.xml. File deps would
> still be supported, because external packages and users want to use
> them, but they would not be allowed for distro packages.
> 
> Not downloading or loading filelists.xml which are required for file
> deps would provide significant bandwidth savings (~47 MB compressed)
> and noticeable runtime savings (~10s at dnf startup) in many common
> cases.
> 
> So this is something that is worth exploring, but it's not clear if
> it
> is at all feasible. It seems that dnf would need to support loading
> filelists.xml lazily. In the mailing list discussions, some people
> said that this would be hard, some people said that it would be
> possible… What is the situation here? IIUC, dnf would need to restart
> the resolution of a transaction mid-flight once it encounters a file
> dep,
> which would require support across the different layers.
> If Fedora commits to making use of this, would it be possible to
> implement this in dnf? What kind of changes would be required?
> 
> [1] https://pagure.io/packaging-committee/issue/714
> [2] https://pagure.io/fesco/issue/1955
> 
> Zbyszek, on behalf of FESCo (but not that this writeup is based
> on my understanding, so all errors are mine.)

This is a bit off on a tangent, but we are working on zchunk (see https
://fedoraproject.org/wiki/Changes/Zchunk_Metadata) which should make
this whole problem far less painful.  As things are going, it's highly
unlikely that we'll be completely done for F29 (we have working code
that needs review), but it should be good to go for F30.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/KJFZE3PHACBQLZKB537LEJODFR3B5FED/

Re: F29 System Wide Change: Zchunk Metadata

[Jonathan Dieter]

On Tue, 2018-07-10 at 08:34 +0100, Jonathan Dieter wrote:
> On Mon, 2018-07-09 at 18:48 -0400, Josh Boyer wrote:
> > Can we update the language on the Change page to clarify that?
> 
> I thought I had explained at:
> https://fedoraproject.org/wiki/Changes/Zchunk_Metadata#Upgrade.2Fcompat
> ibility_impact
> 
> Is there something I should put that would make it clearer?

Ok, I saw your comment on the FESCO ticket, and I see why it's
confusing.

I've updated the summary to say:
"All dnf repository metadata will be compressed with the zchunk format
in addition to xz or gzip."

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/YOREED4D2SW7PQ74LFIMHD2XEV4XJVME/

Re: F29 System Wide Change: Zchunk Metadata

[Jonathan Dieter]

On Mon, 2018-07-09 at 18:48 -0400, Josh Boyer wrote:
> Can we update the language on the Change page to clarify that?

I thought I had explained at:
https://fedoraproject.org/wiki/Changes/Zchunk_Metadata#Upgrade.2Fcompat
ibility_impact

Is there something I should put that would make it clearer?

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/THLHGBEKNOS573J32DDW5HP2AIW5DFN6/

Re: Change DNF to use Zchunk for F29

[Jonathan Dieter]

On Mon, 2018-07-09 at 02:27 +0200, Kevin Kofler wrote:
> Jonathan Dieter wrote:
> > And, since the patches to librepo all work and are in the review state
> > right now, I expect that there will be few, if any, further changes to
> > zchunk's API.
> 
> I think these:
> https://github.com/zchunk/zchunk/blob/8818802cbccde20f911ae527d40037e8fbea36bf/src/zck.c#L209
> https://github.com/zchunk/zchunk/blob/8818802cbccde20f911ae527d40037e8fbea36bf/src/zck.c#L215
> really need to change if you want zchunk to be usable for any other 
> application. The signature that marks the beginning of a chunk ought to be 
> passed through the API rather than hardcoded.

You are absolutely right, and this code will be removed shortly. 
Librepo doesn't actually use this code; it was there so I could
manually compress repodata files for testing purposes before the
librepo code was even written.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/CIRDDFAPQF76STSBQM5W5P7MBNFS3UCO/

Re: Change DNF to use Zchunk for F29

[Jonathan Dieter]

On Mon, 2018-07-02 at 17:07 +0100, Igor Gnatenko wrote:
> Ah, you mean this. So the problem here is that there is no
> alternative to this here. There is casync, but it is not designed for
> this type of usage. Zchunk is good, but it's really dead.

I think you mean "zsync is good, but it's really dead."

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/AQMTEVJZKLBQ4AZ3WZWP6V46SYYAEUUK/

Re: Change DNF to use Zchunk for F29

[Jonathan Dieter]

On Mon, 2018-07-02 at 07:46 -0700, Gerald B. Cox wrote:
> https://fedoraproject.org/wiki/Changes/Zchunk_Metadata
> 
> It appears to be a good idea, but when going through the readme, I
> found this:
> 
> Please note that, while the code is pretty reliable and the file
> format shouldn't see any further changes, the API is still not fixed.
> Please do not use zchunk for any mission-critical systems yet.
> 
> I would consider DNF to be a mission critical system.  Shouldn't we
> wait until zchunk is deemed
> ready?  I don't understand how it is OK to use this for DNF, but it
> isn't OK for
> "any mission-critical systems".

Sorry, this is my fault.  Zchunk was designed with the use-case of dnf
repository metadata in mind.  I put that line in the README to make
sure that nobody used zchunk for unrelated projects until we'd
stabilized the API by actually using it in librepo and friends (what
dnf uses to actually download its metadata).

In other words, this feature is driving zchunk's design rather than the
other way around.

And, since the patches to librepo all work and are in the review state
right now, I expect that there will be few, if any, further changes to
zchunk's API.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/GS23ZJTXC2HZMPV7R6BODWJRC33KWYTW/

Re: Packages with scriptlets which call ldconfig

[Jonathan Dieter]

On Wed, 2018-06-27 at 20:30 -0500, Jason L Tibbitts III wrote:
> usbipjdieter

Fixed in git for F28+

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ANOVOB3GH26OIF5RH6M2FHVZ27A5MGVT/

Old deltarpms are being thrown away on each compose (was Re: dnf and deltarpm)

[Jonathan Dieter]

On Thu, 2018-05-31 at 22:34 +0100, Tomasz Kłoczko wrote:
> Just checked on few mirrors usual location of f28 updates
> (/pub/linux/dist/fedora/linux/updates/28/Everything/x86_64/drpms) and
> in this directory there are at the moment only 56 files from May 31
> and nothing older. So not two drpm per RPM package but only generated
> files out of last batch of updates.

(CC'ing the Fedora infrastructure list)

This is a bug.  It looks like we're throwing away any drpms not
generated in this compose, when we should be keeping them.

I've just created:
https://pagure.io/fedora-infrastructure/issue/7008

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DSHBJB2D3ZSXMSFIZY7MZELKSI6PUAMA/

Re: Status of OwnCloud/NextCloud

[Jonathan Dieter]

On Tue, 2018-04-03 at 19:00 +, Christian Glombek wrote:
> And how many people actually still run NC v10?

FWIW, I have a nextcloud 10 setup on a Fedora 27 server.

I have no problems with major version updates within a Fedora release,
but my expectation was that Fedora was keeping it updated.

If I have to jump through some hoops to upgrade from 10 to 13, that's
not a problem.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Announcing DNF 3 development

[Jonathan Dieter]

On Mon, 2018-03-26 at 11:44 +0300, Jonathan Dieter wrote:
> I've been working on zchunk which will allow downloading only the delta
>  of the metadata.

For reference, the proposal starts at http://lists.rpm.org/pipermail/rp
m-ecosystem/2018-February/000534.html.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Announcing DNF 3 development

[Jonathan Dieter]

On Mon, 2018-03-26 at 09:30 +0100, Richard Hughes wrote:
> With version 3.0 dnf is switching to the codebase we originally wrote
> for PackageKit (libzif->libhif->libdnf), and so it inherits the
> "download complete cache and switch atomically" logic. This means we
> get a shared cache for free.

I've been working on zchunk which will allow downloading only the delta
 of the metadata.  AIUI, currently dnf uses librepo to download the
metadata, so that's where I was planning to do the zchunk integration. 
Is there somewhere else I should be looking at for dnf 3.0, or is it
still using librepo to do the actual downloads?

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: [ACTION NEEDED #2] Missing BuildRequires: gcc/gcc-c++

[Jonathan Dieter]

On Wed, 2018-03-07 at 08:43 +0100, Igor Gnatenko wrote:
> This is the second iteration of my mass-scratch-rebuild without
> gcc/gcc-c++ in the buildroot[0]. Everything what was written in
> original mail still applies.


I've done lizardfs, naev, novacom-client, and novacom-server.

Jonathan

signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: [HEADS UP] Removal of systemd-units

[Jonathan Dieter]

On Thu, 2018-01-25 at 12:17 -0600, Jason L Tibbitts III wrote:
> > > > > > "IG" == Igor Gnatenko 
> > > > > > writes:
> 
> IG> It's just 198 packages and we are just going to fix them for you
> in
> IG> following days.
> 
> jdieterlizardfs novacom-server

Both lizardfs and novacom-server are now fixed in git.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: LizardFS NFS Ganesha integration with no shared library available

[Jonathan Dieter]

On Thu, 2018-01-18 at 07:34 -0500, Kaleb S. KEITHLEY wrote:
> On 01/18/2018 03:23 AM, Jonathan Dieter wrote:
> > What is the best way for me to include NFS Ganesha support in LizardFS?
> >1. Include the latest Fedora NFS Ganesha source and add a hard requires
> >   to that version in the LizardFS NFS subpackage.
> >2. Convince the LizardFS developers to try to move their FSAL into NFS
> >   Ganesha? (LizardFS has just added a client library and doesn't have
> >   a server library, so this will probably take some work)
> 
> That would be the path of least resistance. And the one I would suggest
> and highly recommend. If they've already written a FSAL it should be
> easy to add it to the nfs-ganesha source.
> 
> Depending on the license of course.

If this is both the easiest and the recommended path, then lets see if
the LizardFS developers will go for it.  The license for this code is
GPLv3.  

> >3. Convince the NFS Ganesha developers to create a library that you can
> >   compile FSAL's against?  (The last thread I saw in the NFS Ganesha
> >   devel list on this subject was six years old)
> 
> That would be a non-trivial amount of work with little benefit to the
> current FSAL developers.
> 
> It's not an unreasonable request, per se, but we have other, higher
> priorities.

I totally understand that.

> >4. ???
> > 
> > Any advice would be great, as I'd love to get this feature into
> > Fedora's release of LizardFS.
> 
> Do you have a contact for someone at LizardFS?  I looked at their web
> site and nothing popped out at me.

The main developer that I've communicated with is:
Piotr Sarna <sa...@skytechnology.pl>

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

LizardFS NFS Ganesha integration with no shared library available

[Jonathan Dieter]

(This is meant for the Fedora devel mailing list, but I've cc'd the
nfs-ganesha mailing list in the hopes that they might see something I'm
missing)

The latest release of the LizardFS distributed filesystem includes a
FSAL for NFS Ganesha, allowing you to mount a LizardFS filesystem using
NFS (or pNFS, if you prefer).

Unfortunately, NFS Ganesha doesn't provide a library to build the
LizardFS FSAL against, so LizardFS upstream downloads the complete NFS
Ganesha source and builds against that.

As far as I can see, all the other FSAL's are maintained and built in
NFS Ganesha.

What is the best way for me to include NFS Ganesha support in LizardFS?
   1. Include the latest Fedora NFS Ganesha source and add a hard requires
  to that version in the LizardFS NFS subpackage.
   2. Convince the LizardFS developers to try to move their FSAL into NFS
  Ganesha? (LizardFS has just added a client library and doesn't have
  a server library, so this will probably take some work)
   3. Convince the NFS Ganesha developers to create a library that you can
  compile FSAL's against?  (The last thread I saw in the NFS Ganesha
  devel list on this subject was six years old)
   4. ???

Any advice would be great, as I'd love to get this feature into
Fedora's release of LizardFS.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Security updates and batched pushes

[Jonathan Dieter]

On Wed, 2018-01-10 at 08:06 -0800, Kevin Fenzi wrote:
> On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
> > BTW, are there technical reasons why the metadata is updated
> > en bloc and not incrementally like for example delta RPMs,
> > or is it just that nobody bothered to implement something
> > like that yet for metadata?
> 
> There is no implementation that I know of. It's been talked about for
> many years, but not been implemented.

We talked a bit about using casync for delta metadata at Flock last
summer, and I even did a small proof of concept to see what kind of
savings we'd see, but there were a couple of issues that came up:

 * casync creates a large number of very small files (roughly 4k files
   for yesterday's metadata) that mirrors would need to sync
 * Users would need to download a significant portion of those files to
   delta against (anywhere from 0 to all of them, depending on how
   similar their metadata is)

Downloading any significant number of tiny files over http/1.x is
really, really slow as the downloads are serial.

On reflection, one idea that might fix both problems would be if casync
could concatenate the cacnk files into one large block (cablk,
perhaps), and have the caidx file contain the location of each cacnk in
the block file.  Using http ranges, you would then be able to download
just the parts of the file you wanted (though I have not actually
tested whether specifying a large list of http ranges will download
faster than serially downloading each individual file).

FWIW, casyncing Jan 4th's metadata over Dec 15th's downloads 9.6MB (in
just over 1k cacnk files), while downloading the same metadata directly
would total 16MB.  More frequent updates would be smaller, but I'm not
convinced the difference from an average updates push to another would
be less than 2MB.

Jonathan

signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F28 System Wide Change: AArch64 Server Promotion

[Jonathan Dieter]

On Mon, 2018-01-08 at 12:53 -0500, Stephen John Smoogen wrote:
> On 8 January 2018 at 12:28, Matthew Miller 
> wrote:
> > On Mon, Jan 08, 2018 at 09:16:05AM -0800, Adam Williamson wrote:
> > > A significant miss here is 'testing'. Making an arch primary
> > > means we
> > > need to ensure we have the necessary resources to run all the
> > > relevant
> > > validation testing.
> > 
> > Are there hardware needs here? (Like, not in the server room but in
> > QA
> > team member's hands?)
> > 
> 
> I would say in both. We would need to make sure we have enough
> systems
> which openqa can reliably run on and we need to have some sort of
> system that testers can get a hand on. That would require a bit of a
> 'we expect this hardware to work and this is how you buy/get it' from
> the aarch64 team.. otherwise most everyone is going to come and ask
> why the raspberry pi 3 isn't supported (just like they did with the
> raspberry pi 1 and 2.) [I am not looking to rehash why it isn't ..
> more that is what most testers can get their hands on easily.. and it
> is not a aarch64 platform we support).

I may be going off in the weeds here, but is https://fedoraproject.org/
wiki/Architectures/ARM/Raspberry_Pi#Raspberry_Pi_3_aarch64_support not
accurate when it says it *is* supported?

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Missing deltarpms?

[Jonathan Dieter]

I may have missed something, but I think there might be a problem with
deltarpms in all the current Fedora releases.

Looking at http://mirrors.kernel.org/fedora/updates/27/x86_64/drpms,
there are eight available deltarpms for F27, and if you go to F26,
there are two.

At a guess, new deltarpms are being generated for each push, but the
old ones aren't being copied in anymore.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Rawhide: where for art thou? (why no rawhide composes recently)

[Jonathan Dieter]

On Wed, 2017-06-14 at 09:36 -0600, Kevin Fenzi wrote:
> * Composes are really slow (likely related to storage slowness), if they
> were faster or could fail faster we could untag/fix/iterate more. Right
> now we are lucky to get 2 chances a day.

When you're mentioning storage slowness, are we talking a distributed
filesystem?  If so, is this an issue with metadata lookups being slow
or is it slow data access?

The only reason I ask is that we switched over to LizardFS because its
metadata lookups were *much* faster than what I'd seen from any of the
alternatives.

Jonathan

signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Review Swaps

[Jonathan Dieter]

On Thu, 2017-04-13 at 11:24 -0500, Greg Hellings wrote:
> I have a trio of reviews looking for reviewers. I'll be happy to swap
> for them. Two Python libraries, and an app that depends on them:
> 
> python-camel: https://bugzilla.redhat.com/show_bug.cgi?id=1441841
> python-yamlordereddictloader: https://bugzilla.redhat.com/show_bug.cg
> i?id=1441842
> linchpin: https://bugzilla.redhat.com/show_bug.cgi?id=1441843

Taken.  Can you please take a look at https://bugzilla.redhat.com/show_
bug.cgi?id=1441729?  I'm afraid it's a bit more complex than yours.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Problem building LizardFS - hidden symbol is referenced by DSO

[Jonathan Dieter]

On Wed, 2017-04-12 at 08:02 +0200, Mattia Verga wrote:
> Il 11/04/2017 20:34, Jonathan Dieter ha scritto:
> > I would like to get the LizardFS distributed filesystem into
> > Fedora,
> > but I'm running into problems compiling it with Fedora's hardening
> > flags enabled.

> > I would greatly appreciate any help pointing me in the right
> > direction.
> > 
> > 
> 
> You can try to troubleshoot what flag exactly is causing the problem:
> https://fedoraproject.org/wiki/Changes/Harden_All_Packages#Troublesho
> oting_steps_for_package_maintainers

Thanks, this put me on the right track for debugging it

> Also, here it is an explanation of the error message:
> http://stackoverflow.com/questions/23696585/what-does-exactly-the-
> warning-mean-about-hidden-symbol-being-referenced-by-dso

And this finally helped me figure out what the problem was.

The problem had nothing to do with the hardening flags at all.  For
each binary, LizardFS normally builds a static library that then gets
compiled into the binary.

Fedora's default build flags call for dynamic libraries everywhere, but
that's what caused the bug I was seeing (and doesn't make much sense
anyway, given the context: mfsmetalogger opening the library
mfsmetalogger.so to run?)

Because LizardFS doesn't actually provide any libraries, I've just
passed -DBUILD_SHARED_LIBS:BOOL=OFF and it's fixed.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Problem building LizardFS - hidden symbol is referenced by DSO

[Jonathan Dieter]

I would like to get the LizardFS distributed filesystem into Fedora,
but I'm running into problems compiling it with Fedora's hardening
flags enabled.

When the binaries are linked, I get the following error message for
each binary:
/usr/bin/ld: mfsmetalogger: hidden symbol `__cpu_indicator_init' in 
/usr/lib/gcc/x86_64-redhat-linux/6.3.1/libgcc.a(cpuinfo.o) is referenced by DSO
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

I believe this comes from the file src/common/galois_field_encode.cc
where it has different code paths based on the CPU capabilities, but I
have no idea why this message is coming up or what the correct way is
to fix it.

My preliminary SRPM is at https://www.lesbg.com/jdieter/lizardfs-3.10.6
-2.fc25.src.rpm
The build log is at https://www.lesbg.com/jdieter/lizardfs-build.log

I would greatly appreciate any help pointing me in the right direction.

Thanks,
Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: The glvnd + mesa update for F25

[Jonathan Dieter]

On Sun, 2017-02-05 at 09:10 +0100, Hans de Goede wrote:
> This also mostly explains the why of this change,
> except for why also bring it to Fedora 25 and not
> just to Fedora 26 and later?
> 
> The main reason for this is a non-technical reason,
> we (as in the Fedora project) have quite vocally
> publicly promised we would do so:
> https://blogs.gnome.org/uraeus/2016/11/01/discrete-graphics-and-fedor
> a-workstation-25/

FWIW, I'm really glad this is going into Fedora 25, and I'm very
appreciative of the work you've put into this.  I understand the
pushback against this, but I think it's well worth getting into the
Fedora updates, especially if we don't get any more bug reports.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: GStreamer zero day, round 2...

[Jonathan Dieter]

On Thu, 2016-12-15 at 19:41 -0600, Michael Catanzaro wrote:
> It's not actually round 2, but more like round 5 or something,
> because
> Chris has been publishing an awful lot of these:
> 
> https://scarybeastsecurity.blogspot.com/2016/12/redux-compromising-li
> nux-using-snes.html

There's a new upstream release of game-music-emu that fixes this
particular bug, with a bugzilla report at:

https://bugzilla.redhat.com/show_bug.cgi?id=1405024

I've attached a patch to the bug that will actually allow the new
release to build.

I'm off to push this build to my systems at my school.  Don't want any
of my students getting any bright ideas.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Multiseat seems broken in Fedora 24

[Jonathan Dieter]

On Thu, 2016-11-03 at 11:24 +0100, Sergio Pascual wrote:
> Hello,
> 
> I have a machine with two identical nvidia cards and I configured it
> with multiseat and Fedora 23. It worked well, that is, two GDM login
> screens, one in each display.
> 
> Then I updated to Fedora 24 and it stoped to work.


FWIW, I'm running multiseat systems on F24 using lightdm rather than
GDM (because I want to use xscreensaver and GDM doesn't support that
anymore).  They login to Gnome just fine.

I'm sure filing a bug report, as Ray suggested, is the best solution,
but, if you're looking for a workaround, try using lightdm.

Jonathan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Orphaning pykka

[Jonathan Dieter]

I've just orphaned pykka (https://admin.fedoraproject.org/pkgdb/package
/rpms/pykka/) as I'm no longer using it.

Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: intent to retire Scratch

[Jonathan Dieter]

On Fri, 2016-01-29 at 13:51 -0500, Matthew Miller wrote:
> Several years ago, I packaged up Scratch 1.4 -- the visual
> programming
> language for kids. However, it never really worked perfectly (it's
> not
> 64 bit clean) and upstream for this line is dead (as Scratch 2.0 is
> based on Adobe Air -- and hopefully _soon_ HTML5).
> 
> I'd like to let go of ownership of the package, and unless someone
> else
> really wants to invest time into it, I think retiring it completely
> is
> the way to go.

FWIW, I'm using Scratch on Fedora to teach programming at our school
(based on your recommendation at DevConf 2014), but we're currently
using a custom RPM that contains Scratch 2.0, the Adobe Air version.
1.4 was just too buggy (and was also missing some features that kids
were using on Scratch 2.0 on their Windows systems at home).

I'd say, "Let it go."  And then burst into one of my daughters'
favorite songs.

Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F23 System Wide Change: Glibc locale subpackaging

[Jonathan Dieter]

On Mon, 2015-06-22 at 06:16 -0400, Jan Kurik wrote:
 Recently we made it possible to install a small number of locales by 
 supplying the rpm-macro “_install_langs”, for example
 
rpm -i -D _install_langs=en:de_DE glibc-common.rpm
 
 will install all English locales and all German locales which start 
 with “de_DE”,
 
rpm -i -D _install_langs=en_US.utf8 glibc-common.rpm
 
 will install only the en_US.utf8 locale,
 
rpm -i -D _install_langs=POSIX glibc-common.rpm
 
 will install nothing (but the POSIX/C is still available because it 
 is builtin into glibc).

Please note that this step in implementing the feature will break
deltarpms. Deltarpm requires the *full* data from the original rpm to
be installed in order to build the updated rpm from the deltarpm, so
the fact that we're only installing part of glibc-common will cause any
deltarpms for glibc-common to fail.

If I understand correctly though, the above is a stepping-stone to
subpackages containing the languages, which will *not* break deltarpms.

I do think the change is a great idea; I just want to make sure all
involved are aware of this potential pitfall along the way.

Jonathan
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

USB/IP userspace review request

[Jonathan Dieter]

If anyone is interested in having USB/IP working in Fedora, I'm looking 
for someone to review the userspace tools at 
https://bugzilla.redhat.com/show_bug.cgi?id=1175270.


While the USB/IP modules were in staging, this was in RPM Fusion, but 
the modules have been moved out of staging and into the proper kernel, 
so it's been removed from RPM Fusion.  The latest F21 and Rawhide 
kernels do have the modules enabled and available in 
kernel-modules-extra (thanks Josh!), so all that's left is this review.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: No more deltarpms by default

[Jonathan Dieter]

FWIW, I wrote and maintained yum-presto before it was integrated into 
yum.  I've commented inline:


On 10/06/2014 06:31 PM, Kevin Fenzi wrote:

On Mon, 06 Oct 2014 09:07:33 -0400
Stephen Gallagher sgall...@redhat.com wrote:


The deltarpms were meant to serve two purposes

1) (lesser) Address the needs of users in developing countries (where
Fedora is fairly popular) and bandwidth concerns are very
considerable. Many of these users have connections that are either
metered or extremely slow, so deltarpms provides a way to get the
data to them more economically. This of course can be handled with a
non-default option, so we can talk about making that more
discoverable if we disable them by default.


When I wrote yum-presto it was for this reason.  We were on a lousy link 
and I couldn't run a local mirror without deltarpms.  Having said that, 
with download caps now being a thing in certain developed countries, 
this is no longer just a developing countries problem.



2) (Major) Deltarpms significantly (Kevin, can you comment with
numbers?) reduces the load on the Fedora update servers and mirrors,
thereby reducing hosting costs as well as increasing the efficiency of
the available bandwidth (since smaller downloads mean you will be
tying up the line for a shorter amount of time).


I don't recall this ever being a reason. (I could be wrong).

I think this actually is worse for mirrors in some ways. They have to
mirror a bunch more files and take up space (which is very dear to
mirrors). It does get them less BW used, but most mirrors are on big
links and don't really notice.

It's definitely worse for releng. Generating drpms takes a long time
and delays things like updates pushes or composes.


+1


It would be nice to see if we can find ways to improve the performance
of the deltarpm reconstruction instead. Much of the time is spent on
compression/decompression tasks which *should* be massively parallel;
we should be able to speed this up by taking advantage of additional
cores (and hyperthreading) on the system, for example. Another option
might be not to bother recompressing the RPMs but instead just
install an uncompressed RPM (and possibly recompress it out of band
if we needed to keep it in the cache).


Folks wanting to do so are welcome to help out...

Get to coding. ;)


As mentioned elsewhere, the problem *is* signatures.  yum (quite 
rightly) refuses to install an rpm whose signature doesn't match the one 
in the primary repodata.  And I believe that the signature in the RPM is 
also over the whole compressed rpm.  To make this work, we'd need to add 
an uncompressed signature for every package to the primary repodata as 
well as probably the rpms themselves.


We have already written the code to have yum run applydeltarpm in 
parallel according to the number of processors on a system, but it 
remains a single-threaded application that spends most of its time 
recompressing the data.


Finally, if we do want to stop using deltarpms by default, I think the 
easiest thing to do would be to set dnf to use deltarpms if deltarpm is 
not installed, remove the deltarpm requirement that dnf has, and remove 
deltarpm from default installs.  Then upgrades don't get unexpected 
changes in behavior and new installs can be told that getting deltarpm 
support is as easy as dnf install deltarpm.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: No more deltarpms by default

[Jonathan Dieter]

On 10/06/2014 08:57 PM, Reindl Harald wrote:

Am 06.10.2014 um 19:45 schrieb Florian Festi:

The way of getting around all this unnecessary computation is
establishing trust via the deltarpm itself and giving up the idea of
reconstructing the originally rpm as a prove of everything worked out
just fine.

To save even more time the delta would need to be applied directly on
disk without creating an package at all. This would require a deeper
integration with rpm and requires some tricky algorithms to make sure
everything is ok in advance and won't blow up during the rpm
transaction. So if anyone needs a hobby...


oh no - don't tie all together for reasons which did not destory the
world over years - it is a damned good design that the part dealing with
rpm packages don't need to know anything aboutt delta rpms because the
normal packages are created before that step

don't break the unix-way of work the current behavior follows for no
good reason and there is none - otherwise deltarpm would not have been
default over years the way it works now


Ok, granted, this sounds pretty scary.  But if we give rpm the ability 
to upgrade an installed package with a deltarpm, it wouldn't take away 
deltarpm's ability to generate a full rpm from a deltarpm.  And it does 
have the advantage of cutting right through the knot.  We already store 
checksums of the deltarpms in prestodelta.xml, as well as in the 
deltarpm itself.


Probably the biggest weakness would be the chance that something would 
change on-disk between the check stage and actual install stage.  We'd 
have to evaluate whether it's worth making a temporary copy of the old 
data during the check stage and then applying the deltarpm to that.


All of this would require a lot of buy-in from the rpm guys, though 
(Florian, you're one of them, right?).  If I recall correctly, when we 
first looked at deltarpms, one of the selling points was that rpm didn't 
have to change at all.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: havege in polarssl not enabled and maintainer refuses to enable it (#1069394)

[Jonathan Dieter]

On 10/01/2014 03:33 PM, Matthew Miller wrote:

On Wed, Oct 01, 2014 at 08:52:03AM +0300, Jonathan Dieter wrote:

The havege functions in the polarssl package are currently disabled
in the Fedora package.  Newer releases of dolphin-emu, which are in
a popular external repository, require these functions.

According to https://bugzilla.redhat.com/show_bug.cgi?id=1069394#c1,
the HAVEGE feature is disabled because it's controversial and
would lead to security problems, but the maintainer hasn't given
any more explanation than that in the bug report.

Is there any way we can get a second opinion on this?  The external


Yes there is. Since the objection is potentially security related, it would
be good to get the input of the Fedora Security Team (probably on the
security@ mailing list). Second, having had that conversation, if it still
goes nowhere, file a ticket with FESCo.


Thanks Matthew for the roadmap on this.  When doing further research to 
try to work out where dolphin-emu was actually using the code, I found 
that since dolphin-emu's latest release, they've switched to using 
polarssl without the havege functions.  I'm hoping we can backport those 
commits into the latest release.


So, at this point, I think I'm going to desubmit (is that a word? 
unsubmit?) my request for a second opinion and won't be pursuing this 
any further.  Apologies for any wasted time, and thanks Nikos for 
explaining what havege is and why it shouldn't be used in this context.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

havege in polarssl not enabled and maintainer refuses to enable it (#1069394)

[Jonathan Dieter]

The havege functions in the polarssl package are currently disabled in 
the Fedora package.  Newer releases of dolphin-emu, which are in a 
popular external repository, require these functions.


According to https://bugzilla.redhat.com/show_bug.cgi?id=1069394#c1, the 
HAVEGE feature is disabled because it's controversial and would lead 
to security problems, but the maintainer hasn't given any more 
explanation than that in the bug report.


Is there any way we can get a second opinion on this?  The external 
repository follows Fedora's guidelines to the best of their ability, and 
this includes the prohibition on bundling, so we'd really like to get 
this fixed.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: delta rpms - can we turn them off

[Jonathan Dieter]

This has already happened once before a few years back.  IIRC, we 
updated xz on the builder to match the one in Fedora, but our users had 
broken deltarpms until they got the updated xz.


Jonathan

On 07/02/2014 07:46 AM, Richard W.M. Jones wrote:

Doesn't the current process assume that xz always produces the same
output?

What would happen if a newer version of xz/lzma comes out which (for
example) produces better compressed output while still remaining
compatible with the file format and older decompression tools?

Rich.



--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: delta rpms - can we turn them off

[Jonathan Dieter]

On 06/29/2014 04:36 AM, Florian Weimer wrote:

On 06/29/2014 12:32 PM, drago01 wrote:

On Sun, Jun 29, 2014 at 1:55 AM, Jonathan Dieter jdie...@lesbg.com
wrote:


2. RPM would also need to support signatures across the uncompressed
payload
as well as the compressed payload.


Well Florian said that only the header is actually signed not the
payload. So this shouldn't be necessary.


I missed that the information that the payload is XZ-compressed is
likely signed (hard to tell because the current RPM format isn't
documented).  So we'd need a fake XZ implementation that produces an
essentially uncompressed data stream (xz -0 still compresses).

In the meantime, we could try to reduce the compression level to 0
unconditionally in applydeltarpm.


We can do this, but createrepo would need to store the checksum of 0 
level compressed xz rpms in primary, which involve making createrepo 
decompress each rpm and then recompress at level 0.  Not sure what the 
infra folks think about this.


I may run some tests over the next few weeks to see what implementing 
this would actually take.  I am on a two-month vacation, so no promises 
on how quick I'll be.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: delta rpms - can we turn them off

[Jonathan Dieter]

On 06/28/2014 10:56 AM, drago01 wrote:

On Sat, Jun 28, 2014 at 6:54 PM, Orcan Ogetbil oget.fed...@gmail.com wrote:

On Sat, Jun 28, 2014 at 11:49 AM, Kevin Fenzi wrote:


So, sure, we could sign drpms and yum/dnf could check that, but they
still need to assemble the final rpm in order to pass it to rpm.



The question, to my understanding, which may be wrong, is whether the
contents of the assembled rpm need to be compressed or not.


Yeah its about the rpm payload compression.


Sorry for jumping into this so late; we've just flown back to the States 
for a vacation, so I've been off the net while we were traveling.


A few years back I looked at what it would take to skip the 
recompression step of deltarpm while still verifying signatures.  IIRC, 
the following would need to be done:


1. Yum/DNF repodata would need to include uncompressed checksums as well 
as compressed checksums.


2. RPM would also need to support signatures across the uncompressed 
payload as well as the compressed payload.


3. Deltarpm would need to be patched to build non-compressed rpms when 
rebuilding the deltarpms.


As the maintainer of deltarpm, I can easily do (3), but there didn't 
seem to be much buy-in for (1) and (2) the last time I suggested it (no 
references; I can't seem to find the previous thread in the archives).


Please note that this would severely reduce the need for CPU during the 
deltarpm rebuild process, but at the expense of increasing IO as we're 
writing an uncompressed RPM.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: delta repo data - FTL

[Jonathan Dieter]

On 06/24/2014 03:23 PM, Tomas Mlcoch wrote:

Hi poma,

the short answer is no.

The idea (to have deltas between two repodata) and purpose (be able to gen and 
apply such deltas) is the same, but the used techniques and ideas are different.

snip

So, what will it take to get this into Fedora?  Does DNF/yum already 
have the necessary code?  Or is this available as a plugin?  Has the 
deltarepo generator been packaged for Fedora?  Can createrepo be 
extended to call it at the same time that it generates deltarpms?


I am volunteering to review whatever needs to be reviewed.

Jonathan



--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Orphaning drgeo, drgeo-doc

[Jonathan Dieter]

I'm orphaning drgeo and drgeo-doc.  I originally picked them up because 
I thought our school would use drgeo, but we haven't, so I'm letting it go.


drgeo did FTBFS in the latest rebuild, and will probably need some TLC 
to get it building again.  If you decide to take this, please note that 
upstream has moved to a squeak VM image, and I haven't been able to find 
the source for it, so we're still on drgeo 1.x.


Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Major release number bump is lower than beta for html5lib module.

[Jonathan Dieter]

On Sat, 2014-03-01 at 15:39 +0100, Reindl Harald wrote:
 Am 01.03.2014 15:36, schrieb Praveen Kumar:
  Recently Dan filled bug[0] against html5lib[1] module about new
  upstream release but upstream put major version 0.999 which is lower
  that it's beta version 1.0b3.
  
  Now If I update spec file according to upstream release version should
  yum able to identify that 0.999  1.0b3? or should I go ahead and make
  change as Dan suggested to use version 1.0b3-0.999?
  
  [0] https://bugzilla.redhat.com/show_bug.cgi?id=1070082
  [1] https://pypi.python.org/pypi/html5lib
 
 that's why Epoch exists

I agree that the best thing to do is update the epoch, but only because
the original versioning wasn't done according to package guidelines.

If you read
https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Pre-Release_packages, 
you'll note that the version for upstream's 1.0b3 should have been 1.0, and the 
release should have been something like 0.1.b3.  If that had been followed, you 
could easily have made upstream's 0.999 keep version 1.0, and set the release 
to 0.2.999 or something like that.

However, given that these guidelines weren't followed, I think bumping
epoch, changing the version to 0.999 and the release to 1 is probably
your best bet.

Jonathan

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: dnf-0.4.13

[Jonathan Dieter]

On Thu, 2014-02-06 at 17:44 +0100, Ales Kozumplik wrote:
 dnf-0.4.13 is out [1], [2]. F20 version will follow shortly. We ship 
 Delta RPM support, bash completion and keepcache again in this version.
 
 Remember to come meet the team at DevConf.cz this weekend.

Sorry I missed you guys at DevConf.cz.  Thanks so much for getting
deltarpm support into dnf.  It's been the last major blocker for me
personally.

Thanks again,
Jonathan

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: SELinux RPM scriplet issue annoucement

[Jonathan Dieter]

On Sun, 2014-01-19 at 19:15 +0100, drago01 wrote:
 On Sat, Jan 18, 2014 at 9:47 PM, Rahul Sundaram methe...@gmail.com wrote:
  Hi
 
  Since updates don't automatically fix the issue created by
  https://bugzilla.redhat.com/show_bug.cgi?id=1054350 and users are required
  to run a set of steps as a workaround,  shouldn't this be announced via the
  fedora announce list and posted in the Fedora website prominently as well?
 
 So it happened .. how do we prevent it in the future? How did it pass testing?

Should we modify rpm so scriptlet failures aren't fatal?  This is the
second time in the last six months or so that I've seen scriptlet
failures cause major update problems in Fedora, solely because they are
fatal (see https://bugzilla.redhat.com/show_bug.cgi?id=989145).

If scriptlet failures weren't fatal, we wouldn't have the problem we
have now with duplicate packages.  We could have just pushed the selinux
update, then bumped the release for all updates in the last few pushes,
and then rebuilt them.  If some of the scriptlets were vitally
important, they would get run with the new updates.

Am I missing something?  Is there any reason why scriptlet failures
should be fatal?

Jonathan

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: SELinux RPM scriplet issue annoucement

[Jonathan Dieter]

On Jan 19, 2014 8:57 PM, Michael Schwendt mschwe...@gmail.com wrote:

 On Sun, 19 Jan 2014 20:32:26 +0200, Jonathan Dieter wrote:

  If scriptlet failures weren't fatal, we wouldn't have the problem we
  have now with duplicate packages.  We could have just pushed the selinux
  update,

 After installing the previous bad update that breaks scriptlets, how would
 you activate the new selinux policy within the fixed package's %post
scriptlet?
 Instead of updating to the package in permissive mode, you would need to
 run the scriptlet contents manually *and* still reinstall any package were
 the scriptlets failed.

I was focusing on the fact that scriptlet failures lead to duplicates in
the rpm database, but, you're right, it's not the main problem.

I still think there's a good case for making scriptlet errors non - fatal,
but, in this situation, it would have had a minimal benefit.

  [...] then bumped the release for all updates in the last few pushes,
  and then rebuilt them.

 How do you know which packages a user has tried to install/update _after_
 updating to the bad policy package? It could be any package within the
package
 collection that would remain installed but broken because of the
scriptlets bug.
 You assume that users have only applied the few updates following the bad
 selinux policy update.

ACK. I didn't think this part through properly.

Jonathan
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Introducing of Mosaab Alzoubi

[Jonathan Dieter]

On Wed, 2013-10-02 at 12:53 +0200, مصعب الزعبي wrote:
 Hi ,
 
 For instructions here : 
 https://fedoraproject.org/wiki/Join_the_package_collection_maintainers
 I am glad to Introduce myself to you , 
 
 My name Mosaab Alzoubi , 24 years old.
snip

Ahlan wa-sahlan.  If you're ever in Lebanon, look me up.

Jonathan




-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Wider feedback requested on two changes to our base/core defaults

[Jonathan Dieter]

On Thu, 2013-08-22 at 10:39 +, Jóhann B. Guðmundsson wrote:
 On 08/22/2013 09:10 AM, Ondrej Vasik wrote:
  On Wed, 2013-08-21 at 18:45 +, Jóhann B. Guðmundsson wrote:
 snip
  I perfectly understand the reasons for the change and I think we should
  definitely change it at least on the login screen (I like the one
  additional line idea from Simo). In the terminal label, full hostname
  might make sense as well. But I don't like the idea for the command line
  PS1 change. Even if I don't have too long FQDN, it will extend my basic
  prompt from 23 to 38 (almost half of 80 chars) on 1 system I use most
  and to 20 to 43 on another one. This is imho too much (so if the final
  decision would be to change \h to \H, I'm going to change the default
  PS1 back on my machine anyway).
  Having hostname as separate line will make cutpaste of command sequence
  from terminal harder to read. I know that many users modify the basic
  PS1 anyway, but IMHO nothing blocks you from having modified PS1 in
  ~/.bashrc (or directly in /etc/skel/).
 
 I think defaulting to long hostname is better and users can always 
 override this to the short one encase they have to long hostnames.

For the general use-case, I think that a long hostname at the prompt is
overkill.  I have no problem with the full hostname always being shown
at the login screen, but does every computer on my network really need
to waste the space to show an extra 15-20 characters every time the
prompt comes up?

Seeing as this feature seems to make most sense in a cloud setting,
wouldn't it make sense to ship a package (bash-prompt-full-fqdn or
something like that) that contains the necessary PS1 adjustments and
make it default for the Cloud group?

Jonathan

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Wider feedback requested on two changes to our base/core defaults

[Jonathan Dieter]

On Thu, 2013-08-22 at 11:09 +, Jóhann B. Guðmundsson wrote:
snip
 You would just overwrite in in your own .bashrc if you have long 
 hostname and they get in your way.

Yes, the same applies in the opposite direction.

 Long hostnames are far more practical for administrators to use then 
 short hostnames have ever been.

I disagree.  As a system administrator of a small network (around 100
Fedora desktops, all on the same domain name), I'm quite happy with
short hostnames.  

 We seriously should be trying to avoid causing unnecessary confusion to 
 administrator by using short hostnames by default and if you are on a 
 large network you end up with the exact same problem as I mentioned earlier,

And I think you've hit the problem right on the head right here.  *If*
you're on a large network with more than one domain name, then long
hostnames make sense.  But if *not*, they just waste space.

If I understand you right, you're arguing that large networks are a big
enough part of our target audience that long hostnames should be the
default.

I'm arguing that we should give long hostnames to spins/groups that
target large networks (i.e. if you install the Cloud Infrastructure
group, you get long hostnames by default), but leave the default for
most spins as short hostnames.

Jonathan

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Call for Bikeshedding: remote auth at install time

[Jonathan Dieter]

On Mon, 2013-06-03 at 18:07 -0700, Adam Williamson wrote:
 Whether this is a blocker or not comes down to a judgement call, because
 it hinges on whether this is a significant inconvenience for a large
 enough number of users. So we need to know from people who use Fedora in
 remote auth environments whether it's a big problem not to be able to
 set it up at install / firstboot time, or whether you'd be okay with
 creating a local user to get through initial-setup and then configuring
 remote auth from that local account.

For me, as a system administrator of a number of labs that all use
Fedora, the main inconvenience is being forced to create a local user at
install time.  I don't mind going through manual configuration to get
LDAP / Kerberos setup on the desktop, but it's a pain to setup a local
user, complete with password that passes the complexity checks, just to
delete it.

Jonathan

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel