Re: [HACKERS] System load consideration before spawning parallel workers

It seems clear that this patch design is not favored by the community, so I'm setting the patch as rejected in the CF app. I think there is interest in managing system resources better, but I don't know what that would look like. -- Peter Eisentraut http://www.2ndQuadrant.com/

Re: [HACKERS] System load consideration before spawning parallel workers

On Fri, Sep 2, 2016 at 3:01 AM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 8/16/16 3:39 AM, Haribabu Kommi wrote: > > Yes, we need to consider many parameters as a system load, not just only > > the CPU. Here I attached a POC patch that implements the CPU load > >

Re: [HACKERS] System load consideration before spawning parallel workers

On Thu, Sep 1, 2016 at 10:01 AM, Peter Eisentraut wrote: > On 8/16/16 3:39 AM, Haribabu Kommi wrote: >> Yes, we need to consider many parameters as a system load, not just only >> the CPU. Here I attached a POC patch that implements the CPU load >> calculation

Re: [HACKERS] System load consideration before spawning parallel workers

On 9/2/16 4:07 PM, Bruce Momjian wrote: Couldn't SQL sessions call a PL/Perl function that could query the OS and set max_parallel_workers_per_gather appropriately? I'd certainly like to see a greater ability to utilize "hooks" without resorting to C. "hooks" in quotes because while some

Re: [HACKERS] System load consideration before spawning parallel workers

On Thu, Sep 1, 2016 at 01:01:35PM -0400, Peter Eisentraut wrote: > Maybe a couple of hooks could be useful to allow people to experiment > with this. But the hooks should be more general, as described above. > But I think a few GUC settings that can be adjusted at run time could be > sufficient

Re: [HACKERS] System load consideration before spawning parallel workers

Gavin Flower writes: > On 02/09/16 04:44, Peter Eisentraut wrote: >> You can try this out by building PostgreSQL this way. Please save your >> work first, because you might have to hard-reboot your system. > Hmm... I've built several versions of pg this way,

Re: [HACKERS] System load consideration before spawning parallel workers

On 02/09/16 05:01, Peter Eisentraut wrote: On 8/16/16 3:39 AM, Haribabu Kommi wrote: [...] All of this seems very platform specific, too. You have Windows-specific code, but the rest seems very Linux-specific. The dstat tool I had never heard of before. There is stuff with cgroups, which I

Re: [HACKERS] System load consideration before spawning parallel workers

On 02/09/16 04:44, Peter Eisentraut wrote: On 8/1/16 2:17 AM, Gavin Flower wrote: Possibly look how make does it with the '-l' flag? '-l 8' don't start more process when load is 8 or greater, works on Linux at least... The problem with that approach is that it takes about a minute for the

Re: [HACKERS] System load consideration before spawning parallel workers

Peter Eisentraut writes: > As I just wrote in another message in this thread, I don't trust system > load metrics very much as a gatekeeper. They are reasonable for > long-term charting to discover trends, but there are numerous potential > problems for using

Re: [HACKERS] System load consideration before spawning parallel workers

On 8/16/16 3:39 AM, Haribabu Kommi wrote: > Yes, we need to consider many parameters as a system load, not just only > the CPU. Here I attached a POC patch that implements the CPU load > calculation and decide the number of workers based on the available CPU > load. The load calculation code is

Re: [HACKERS] System load consideration before spawning parallel workers

On 8/1/16 2:17 AM, Gavin Flower wrote: > Possibly look how make does it with the '-l' flag? > > '-l 8' don't start more process when load is 8 or greater, works on > Linux at least... The problem with that approach is that it takes about a minute for the load averages figures to be updated, by

Re: [HACKERS] System load consideration before spawning parallel workers

On Fri, Aug 5, 2016 at 9:46 AM, Jim Nasby wrote: > On 8/1/16 1:08 AM, Haribabu Kommi wrote: >> >> There are some utilities and functions that are available to calculate the >> current system load, based on the available resources and system load, >> the module can allow

Re: [HACKERS] System load consideration before spawning parallel workers

On 8/1/16 1:08 AM, Haribabu Kommi wrote: There are some utilities and functions that are available to calculate the current system load, based on the available resources and system load, the module can allow the number of parallel workers that can start. In my observation, adding this

Re: [HACKERS] System load consideration before spawning parallel workers

On 01/08/16 18:08, Haribabu Kommi wrote: On Fri, Jul 29, 2016 at 8:48 PM, Amit Kapila wrote: On Fri, Jul 29, 2016 at 11:26 AM, Haribabu Kommi wrote: we observed that spawning the specified number of parallel workers for every query that

Re: [HACKERS] System load consideration before spawning parallel workers

On Fri, Jul 29, 2016 at 8:48 PM, Amit Kapila wrote: > On Fri, Jul 29, 2016 at 11:26 AM, Haribabu Kommi > wrote: >> we observed that spawning the specified number of parallel workers for >> every query that satisfies for parallelism is sometimes

Re: [HACKERS] System load consideration before spawning parallel workers

On Fri, Jul 29, 2016 at 11:26 AM, Haribabu Kommi wrote: > we observed that spawning the specified number of parallel workers for > every query that satisfies for parallelism is sometimes leading to > performance drop compared to improvement during the peak system load >

[HACKERS] System load consideration before spawning parallel workers

we observed that spawning the specified number of parallel workers for every query that satisfies for parallelism is sometimes leading to performance drop compared to improvement during the peak system load with other processes. Adding more processes to the system is leading to more context