Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
@marxin pushed 1 commit. fff307dd41294192560a33cca09e3675a0af925a Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/fec86dfc33a67ce75c0113b64d0ec4fede975eab..fff307dd41294192560a33cca09e3675a0af925a ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
So the answer is that it's stable and reproducible. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-656525918___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
> I haven't checked the zstd implementation, I admit, I just know that this was > the problem with parallel bzip2/xz compression. I've made some experiments for a 100MB big file and the compressed output is still the same. For being sure, I asked the question in https://github.com/facebook/zstd/issues/2238. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-656046924___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
You need to recreate the same bytewise identical compression when applying the deltarpm. The parallel compression implementations tend to partition the data and compress the chunks individually, leading to a different result. I haven't checked the zstd implementation, I admit, I just know that this was the problem with parallel bzip2/xz compression. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-656002046___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
No. How does it break it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655620102___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
You're aware that this breaks delta rpms? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655579498___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
Demo of the used parallel compression for `cvise` package: https://gist.github.com/marxin/5560256dbf0af4c749bd4d56c0e83626 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655495389___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
Port zstd compression to the new API (ZSTD_compressStream2). My motivation is to eventually enable multi-threaded compression which can help packages like Firefox where theres a single huge rpm that is being compressed. Once the change is accepted, the following one-liner will enable parallel compression: ```c ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 4); ``` You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/1303 -- Commit Summary -- * zstd compression: port to the new API. -- File Changes -- M rpmio/rpmio.c (68) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/1303.patch https://github.com/rpm-software-management/rpm/pull/1303.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
