Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
On 05/21/2015 01:39 PM, Andres Freund wrote: On 2015-05-21 11:54:40 -0700, Josh Berkus wrote: This has been talked about as a feature, but would require major work on PostgreSQL to make it possible. You'd be looking at several months of effort by a really good hacker, and then a whole bunch of performance testing. If you have the budget for this, then please let's talk about it because right now nobody is working on it. I think this is overestimating the required effort quite a bit. While not trivial, it's also not that complex to make this work. Well, then, maybe someone should hack and test it ... -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
On 04/23/2015 12:47 PM, Jan Gunnar Dyrset wrote: I think that preallocating lumps of a given, configurable size, say 4 MB, for the tables would remove this problem. The max number of fragments on a 1 GB file would then be 250, which is no problem. Is this possible to configure in PostgreSQL? If not, how difficult is it to implement in the database? It is not currently possible to configure. This has been talked about as a feature, but would require major work on PostgreSQL to make it possible. You'd be looking at several months of effort by a really good hacker, and then a whole bunch of performance testing. If you have the budget for this, then please let's talk about it because right now nobody is working on it. Note that this could be a dead end; it's possible that preallocating large extents could cause worse problems than the current fragmentation issues. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
On 2015-05-21 11:54:40 -0700, Josh Berkus wrote: This has been talked about as a feature, but would require major work on PostgreSQL to make it possible. You'd be looking at several months of effort by a really good hacker, and then a whole bunch of performance testing. If you have the budget for this, then please let's talk about it because right now nobody is working on it. I think this is overestimating the required effort quite a bit. While not trivial, it's also not that complex to make this work. Greetings, Andres Freund -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
It may be even easier. AFAIR, it's possible just to tell OS expected allocation without doing it. This way nothing changes for general code, it's only needed to specify expected file size on creation. Please see FILE_ALLOCATION_INFO: https://msdn.microsoft.com/en-us/library/windows/desktop/aa364214(v=vs.85).aspx Чт, 21 трав. 2015 16:39 Andres Freund and...@anarazel.de пише: On 2015-05-21 11:54:40 -0700, Josh Berkus wrote: This has been talked about as a feature, but would require major work on PostgreSQL to make it possible. You'd be looking at several months of effort by a really good hacker, and then a whole bunch of performance testing. If you have the budget for this, then please let's talk about it because right now nobody is working on it. I think this is overestimating the required effort quite a bit. While not trivial, it's also not that complex to make this work. Greetings, Andres Freund -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
On 2015-04-29 10:06:39 +0200, Andres Freund wrote: Hi, On 2015-04-23 19:47:06 +, Jan Gunnar Dyrset wrote: I am using PostgreSQL to log data in my application. A number of rows are added periodically, but there are no updates or deletes. There are several applications that log to different databases. This causes terrible disk fragmentation which again causes performance degradation when retrieving data from the databases. The table files are getting more than 5 fragments over time (max table size about 1 GB). The problem seems to be that PostgreSQL grows the database with only the room it need for the new data each time it is added. Because several applications are adding data to different databases, the additions are never contiguous. Which OS and filesystem is this done on? Because many halfway modern systems, like e.g ext4 and xfs, implement this in the background as 'delayed allocation'. Oh, it's in the subject. Stupid me, sorry for that. I'd consider testing how much better this behaves under a different operating system, as a shorter term relief. Greetings, Andres Freund -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
Hi, On 2015-04-23 19:47:06 +, Jan Gunnar Dyrset wrote: I am using PostgreSQL to log data in my application. A number of rows are added periodically, but there are no updates or deletes. There are several applications that log to different databases. This causes terrible disk fragmentation which again causes performance degradation when retrieving data from the databases. The table files are getting more than 5 fragments over time (max table size about 1 GB). The problem seems to be that PostgreSQL grows the database with only the room it need for the new data each time it is added. Because several applications are adding data to different databases, the additions are never contiguous. Which OS and filesystem is this done on? Because many halfway modern systems, like e.g ext4 and xfs, implement this in the background as 'delayed allocation'. Is it possible that e.g. you're checkpointing very frequently - which includes fsyncing dirty files - and that that causes delayed allocation not to work? How often did you checkpoint? How did you measure the fragmentation? Using filefrag? If so, could you perhaps send its output? I think that preallocating lumps of a given, configurable size, say 4 MB, for the tables would remove this problem. The max number of fragments on a 1 GB file would then be 250, which is no problem. Is this possible to configure in PostgreSQL? If not, how difficult is it to implement in the database? It's not impossible, but there are complexities because a) extension happens under a sometimes contended lock, and doing more there will have possible negative scalability implications. we need to restructure the logging first to make that more realistic. b) postgres also tries to truncate files, and we need to make sure that happens only in the right cirumstances. Greetings, Andres Freund -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
On 04/29/2015 01:08 AM, Andres Freund wrote: Which OS and filesystem is this done on? Because many halfway modern systems, like e.g ext4 and xfs, implement this in the background as 'delayed allocation'. Oh, it's in the subject. Stupid me, sorry for that. I'd consider testing how much better this behaves under a different operating system, as a shorter term relief. This is a known issue on the Windows platform. It is part of the limitations of that environment. Linux/Solaris/FreeBSD do not suffer from this issue in nearly the same manner. jD -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Centered full stack support, consulting and development. Announcing I'm offended is basically telling the world you can't control your own emotions, so everyone else should do it for you. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
On Wed, Apr 29, 2015 at 07:07:04AM -0700, Joshua D. Drake wrote: On 04/29/2015 01:08 AM, Andres Freund wrote: Which OS and filesystem is this done on? Because many halfway modern systems, like e.g ext4 and xfs, implement this in the background as 'delayed allocation'. Oh, it's in the subject. Stupid me, sorry for that. I'd consider testing how much better this behaves under a different operating system, as a shorter term relief. This is a known issue on the Windows platform. It is part of the limitations of that environment. Linux/Solaris/FreeBSD do not suffer from this issue in nearly the same manner. jD You might consider a CLUSTER or VACUUM FULL to re-write the table with less fragmentation. Regards, Ken -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
On 04/29/2015 10:35 AM, k...@rice.edu wrote: On Wed, Apr 29, 2015 at 07:07:04AM -0700, Joshua D. Drake wrote: On 04/29/2015 01:08 AM, Andres Freund wrote: Which OS and filesystem is this done on? Because many halfway modern systems, like e.g ext4 and xfs, implement this in the background as 'delayed allocation'. Oh, it's in the subject. Stupid me, sorry for that. I'd consider testing how much better this behaves under a different operating system, as a shorter term relief. This is a known issue on the Windows platform. It is part of the limitations of that environment. Linux/Solaris/FreeBSD do not suffer from this issue in nearly the same manner. jD You might consider a CLUSTER or VACUUM FULL to re-write the table with less fragmentation. Or pg_repack if you can't handle the lockup time that these involve. cheers andrew -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
[PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows
I am using PostgreSQL to log data in my application. A number of rows are added periodically, but there are no updates or deletes. There are several applications that log to different databases. This causes terrible disk fragmentation which again causes performance degradation when retrieving data from the databases. The table files are getting more than 5 fragments over time (max table size about 1 GB). The problem seems to be that PostgreSQL grows the database with only the room it need for the new data each time it is added. Because several applications are adding data to different databases, the additions are never contiguous. I think that preallocating lumps of a given, configurable size, say 4 MB, for the tables would remove this problem. The max number of fragments on a 1 GB file would then be 250, which is no problem. Is this possible to configure in PostgreSQL? If not, how difficult is it to implement in the database? Thank you, JG
