Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-30 Thread Panu Matilainen
Merged #2418 into master. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2418#event-8886106505 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-30 Thread Panu Matilainen
Okay, since there are no further comments... -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2418#issuecomment-1490100013 You are receiving this because you are subscribed to this thread. Message ID:

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-27 Thread Michal Domonkos
OK, I can confirm the new push fixes the `proc` calculation, thanks! -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2418#issuecomment-1485189177 You are receiving this because you are subscribed to this thread. Message ID:

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-27 Thread Michal Domonkos
@dmnks commented on this pull request. > +long int pagesize = sysconf(_SC_PAGESIZE); +long int pages = sysconf(_SC_PHYS_PAGES); + +if (pagesize < 0) + pagesize = 4096; +if (pages > 0) + mem = pages * pagesize; + +return mem; +} + +static unsigned long

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-24 Thread Panu Matilainen
@pmatilai commented on this pull request. > +long int pagesize = sysconf(_SC_PAGESIZE); +long int pages = sysconf(_SC_PHYS_PAGES); + +if (pagesize < 0) + pagesize = 4096; +if (pages > 0) + mem = pages * pagesize; + +return mem; +} + +static unsigned long

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-24 Thread Panu Matilainen
@pmatilai commented on this pull request. > + * Conservative estimates for thread use on 32bit systems where address + * space is an issue: 2GB for bare metal, 3GB for a 32bit process + * on a 64bit system. + */ +if (thread) { + unsigned long vmem = mem; +#if

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-24 Thread Panu Matilainen
@pmatilai commented on this pull request. > @@ -1174,6 +1177,89 @@ static void doShescape(MacroBuf mb, rpmMacroEntry me, > ARGV_t argv, size_t *parse mbAppend(mb, '\''); } +static unsigned long getmem_total(void) +{ +unsigned long mem = 0; +long int pagesize =

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-24 Thread Panu Matilainen
@pmatilai pushed 3 commits. c013e36fa47006ef8598a74450e709911a2a0b9d Add optional total/proc/thread arguments to %{getncpus} macro 758e92b4b4d94792387c8999d2a258cd18225020 Make rpmio parallelism memory aware 73dad5c636524d1dec6f22df7de2a6a669ebbc70 Make build parallelism memory and

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-24 Thread Panu Matilainen
@pmatilai commented on this pull request. > + * Conservative estimates for thread use on 32bit systems where address + * space is an issue: 2GB for bare metal, 3GB for a 32bit process + * on a 64bit system. + */ +if (thread) { + unsigned long vmem = mem; +#if

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-22 Thread Michal Domonkos
Other than the above, the overall changeset looks sane to me. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2418#issuecomment-1479932861 You are receiving this because you are subscribed to this thread. Message ID:

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-22 Thread Michal Domonkos
@dmnks commented on this pull request. > +long int pagesize = sysconf(_SC_PAGESIZE); +long int pages = sysconf(_SC_PHYS_PAGES); + +if (pagesize < 0) + pagesize = 4096; +if (pages > 0) + mem = pages * pagesize; + +return mem; +} + +static unsigned long

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-22 Thread Michal Domonkos
@dmnks commented on this pull request. > @@ -1174,6 +1177,89 @@ static void doShescape(MacroBuf mb, rpmMacroEntry me, > ARGV_t argv, size_t *parse mbAppend(mb, '\''); } +static unsigned long getmem_total(void) +{ +unsigned long mem = 0; +long int pagesize =

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-22 Thread Michal Domonkos
@dmnks commented on this pull request. > @@ -1174,6 +1177,89 @@ static void doShescape(MacroBuf mb, rpmMacroEntry me, > ARGV_t argv, size_t *parse mbAppend(mb, '\''); } +static unsigned long getmem_total(void) +{ +unsigned long mem = 0; +long int pagesize =

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-22 Thread Michal Domonkos
@dmnks commented on this pull request. > @@ -1174,6 +1177,89 @@ static void doShescape(MacroBuf mb, rpmMacroEntry me, > ARGV_t argv, size_t *parse mbAppend(mb, '\''); } +static unsigned long getmem_total(void) +{ +unsigned long mem = 0; +long int pagesize =

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-22 Thread Michal Domonkos
@dmnks commented on this pull request. > + * Conservative estimates for thread use on 32bit systems where address + * space is an issue: 2GB for bare metal, 3GB for a 32bit process + * on a 64bit system. + */ +if (thread) { + unsigned long vmem = mem; +#if __WORDSIZE

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-22 Thread Michal Domonkos
@dmnks commented on this pull request. > + * Conservative estimates for thread use on 32bit systems where address + * space is an issue: 2GB for bare metal, 3GB for a 32bit process + * on a 64bit system. + */ +if (thread) { + unsigned long vmem = mem; +#if __WORDSIZE

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-21 Thread Neal Gompa
That's even better.  -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2418#issuecomment-1477683506 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-21 Thread Panu Matilainen
Well, the point of this PR is to make %limit_build and the like redundant entirely. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2418#issuecomment-1477342247 You are receiving this because you are subscribed to this thread. Message

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-20 Thread Neal Gompa
`%getmem` could be useful for making `%limit_build` work better in Fedora, so I think it'd be worth having. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2418#issuecomment-1476902969 You are receiving this because you are subscribed to

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-08 Thread Panu Matilainen
In my original version this also added a %getmem macro with similar arguments. Left it out to keep things simple and minimal, but would be trivial to add back if people think that's useful. -- Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-08 Thread Panu Matilainen
@pmatilai pushed 1 commit. e4d0e62d129569a714ae190b2ffd2fcc1296834d Make build parallelism memory and address-space aware (#804) -- View it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-08 Thread Panu Matilainen
@pmatilai pushed 2 commits. c859ebe35c54e58f689bc3d2097ff0acbe8172fa Make rpmio parallelism memory aware 3353d6a4234dd8e2ead28fc7cffdae9cc68e24b1 Make build parallelism memory and address-space aware (#804) -- View it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-08 Thread Panu Matilainen
@pmatilai pushed 3 commits. 28b7237b30ad02ad2babee4d8abb13c96ae360fa Add optional total/proc/thread arguments to %{getncpus} macro b09677daadfa7025eb2ff89560c0a886e043302c Make rpmio parallelism memory aware 753763a1f971c0e041f934c442337a012eb7df5b Make build parallelism memory and

[Rpm-maint] [rpm-software-management/rpm] Consider available memory and address space for parallel execution (PR #2418)

2023-03-08 Thread Panu Matilainen
See commits for details, but short summary: add optional proc/thread argument to %{getncpus} macro which consider the available memory and address space, based on newly added tunables for tasksize. The goal here is to avoid build failures due to gross overallocation on constrained systems,