Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-10 Thread marxin
@marxin pushed 1 commit. fff307dd41294192560a33cca09e3675a0af925a Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-10 Thread marxin
So the answer is that it's stable and reproducible. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-09 Thread marxin
> I haven't checked the zstd implementation, I admit, I just know that this was > the problem with parallel bzip2/xz compression. I've made some experiments for a 100MB big file and the compressed output is still the same. For being sure, I asked the question in

Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-09 Thread Michael Schroeder
You need to recreate the same bytewise identical compression when applying the deltarpm. The parallel compression implementations tend to partition the data and compress the chunks individually, leading to a different result. I haven't checked the zstd implementation, I admit, I just know that

Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-08 Thread marxin
No. How does it break it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655620102___ Rpm-maint mailing list

Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-08 Thread Michael Schroeder
You're aware that this breaks delta rpms? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655579498___

Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-08 Thread marxin
Demo of the used parallel compression for `cvise` package: https://gist.github.com/marxin/5560256dbf0af4c749bd4d56c0e83626 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)

2020-07-08 Thread marxin
Port zstd compression to the new API (ZSTD_compressStream2). My motivation is to eventually enable multi-threaded compression which can help packages like Firefox where theres a single huge rpm that is being compressed. Once the change is accepted, the following one-liner will enable parallel