Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Mark Kirkwood
On 04/06/15 12:58, Scott Marlowe wrote: On Wed, Jun 3, 2015 at 6:53 PM, Scott Marlowe wrote: On Wed, Jun 3, 2015 at 4:29 PM, Joshua D. Drake wrote: On 06/03/2015 03:16 PM, Tomas Vondra wrote: What is more important, though, is the amount of memory. OP reported the query writes ~95GB of tem

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Scott Marlowe
On Wed, Jun 3, 2015 at 6:53 PM, Scott Marlowe wrote: > On Wed, Jun 3, 2015 at 4:29 PM, Joshua D. Drake > wrote: >> >> On 06/03/2015 03:16 PM, Tomas Vondra wrote: >> >>> What is more important, though, is the amount of memory. OP reported the >>> query writes ~95GB of temp files (and dies because

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Scott Marlowe
On Wed, Jun 3, 2015 at 4:29 PM, Joshua D. Drake wrote: > > On 06/03/2015 03:16 PM, Tomas Vondra wrote: > >> What is more important, though, is the amount of memory. OP reported the >> query writes ~95GB of temp files (and dies because of full disk, so >> there may be more). The on-disk format is u

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Tomas Vondra
On 06/04/15 01:54, Yves Dorfsman wrote: On 2015-06-03 16:29, Joshua D. Drake wrote: On 06/03/2015 03:16 PM, Tomas Vondra wrote: What is more important, though, is the amount of memory. OP reported the query writes ~95GB of temp files (and dies because of full disk, so there may be more). Th

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Yves Dorfsman
On 2015-06-03 16:29, Joshua D. Drake wrote: > > On 06/03/2015 03:16 PM, Tomas Vondra wrote: > >> What is more important, though, is the amount of memory. OP reported the >> query writes ~95GB of temp files (and dies because of full disk, so >> there may be more). The on-disk format is usually mor

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Joshua D. Drake
On 06/03/2015 03:16 PM, Tomas Vondra wrote: What is more important, though, is the amount of memory. OP reported the query writes ~95GB of temp files (and dies because of full disk, so there may be more). The on-disk format is usually more compact than the in-memory representation - for example

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Tomas Vondra
On 06/03/15 23:18, Scott Marlowe wrote: On Wed, Jun 3, 2015 at 1:24 PM, Tomas Vondra wrote: On 06/03/15 17:09, Scott Marlowe wrote: On Wed, Jun 3, 2015 at 8:56 AM, Tomas Vondra Well, except that 15GB of that is shared_buffers, and I wouldn't call that 'free'. Also, I don't see page cache

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Claudio Freire
On Wed, Jun 3, 2015 at 6:18 PM, Scott Marlowe wrote: > On Wed, Jun 3, 2015 at 1:24 PM, Tomas Vondra > wrote: >> >> >> On 06/03/15 17:09, Scott Marlowe wrote: >>> >>> On Wed, Jun 3, 2015 at 8:56 AM, Tomas Vondra I don't see why you think you have less than 3GB used. The output

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Scott Marlowe
On Wed, Jun 3, 2015 at 1:24 PM, Tomas Vondra wrote: > > > On 06/03/15 17:09, Scott Marlowe wrote: >> >> On Wed, Jun 3, 2015 at 8:56 AM, Tomas Vondra >>> >>> >>> >>> I don't see why you think you have less than 3GB used. The output you >>> posted >>> clearly shows there's only ~300MB memory free -

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Tomas Vondra
On 06/03/15 17:09, Scott Marlowe wrote: On Wed, Jun 3, 2015 at 8:56 AM, Tomas Vondra I don't see why you think you have less than 3GB used. The output you posted clearly shows there's only ~300MB memory free - there's 15GB shared buffers and ~45GB of page cache (file system cache). Because

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Claudio Freire
On Wed, Jun 3, 2015 at 11:56 AM, Tomas Vondra wrote: > On 06/03/15 16:06, ben.play wrote: >> >> The query is (unfortunately) generated by Doctrine 2 (Symfony 2). >> We can’t change the query easily. > > > Well, then you'll probably have to buy more RAM, apparently. There's an easy way to add disk

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Scott Marlowe
On Wed, Jun 3, 2015 at 8:56 AM, Tomas Vondra wrote: > > On 06/03/15 16:06, ben.play wrote: >> >> The query is (unfortunately) generated by Doctrine 2 (Symfony 2). >> We can’t change the query easily. > > > Well, then you'll probably have to buy more RAM, apparently. > >> This is my config : >> >>

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Tomas Vondra
On 06/03/15 16:06, ben.play wrote: The query is (unfortunately) generated by Doctrine 2 (Symfony 2). We can’t change the query easily. Well, then you'll probably have to buy more RAM, apparently. This is my config : max_connections = 80 shared_buffers = 15GB work_mem = 384MB maintenance_wor

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread ben.play
The query is (unfortunately) generated by Doctrine 2 (Symfony 2). We can’t change the query easily. This is my config : max_connections = 80 shared_buffers = 15GB work_mem = 384MB maintenance_work_mem = 1GB #temp_buffers = 8MB #temp_file_limit = -1 effective_cache_size = 44GB If I put a temp_

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Tomas Vondra
On 06/03/15 15:27, chiru r wrote: Hi Benjamin, It looks you are facing disk space issue for queries. In order to avid the disk space issue you can do the following. 1) Increase the work_mem parameter session level before executing the queries. 2) If you observe diskspace issue particular user

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread chiru r
Hi Benjamin, It looks you are facing disk space issue for queries. In order to avid the disk space issue you can do the following. 1) Increase the work_mem parameter session level before executing the queries. 2) If you observe diskspace issue particular user queries,increase the work_mem paramete

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread amulsul
SQLSTATE[53100]: Disk full: 7 ERROR: could not write block 1099247 of temporary file Its looks like there is no room to write temporary file, try with limiting temporary file size by setting temp_file_limit GUC. -- View this message in context: http://postgresql.nabble.com/How-to-reduce-writ

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread David G. Johnston
You should repost this directly and not through Nabble. It has wrapped your code in raw tags which the PostgreSQL mailing list software strips. On Wednesday, June 3, 2015, ben.play wrote: > Hi all, > > We have a big database (more than 300 Gb) and we run a lot of queries each > minute. > > Howe

[PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread ben.play
Hi all, We have a big database (more than 300 Gb) and we run a lot of queries each minute. However, once an hour, the (very complex) query writes A LOT on the disk (more than 95 Gb !!!) We have 64 Gb of RAM and this is our config : And my error on the query is : Do you know how to solve th