Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-22 Thread Syed, Rahila
Hello, Please find attached the patch to compress FPW. Sorry I had forgotten to attach. Please find the patch attached. Thank you, Rahila Syed From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Rahila Syed Sent: Monday, September 22, 2014

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-10-28 Thread Syed, Rahila
if fpw=on which is not the case for compressedPages. Thank you, Rahila Syed -Original Message- From: Fujii Masao [mailto:masao.fu...@gmail.com] Sent: Monday, October 27, 2014 6:50 PM To: Rahila Syed Cc: Andres Freund; Syed, Rahila; Alvaro Herrera; Rahila Syed; PostgreSQL-development; Tom Lane

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-26 Thread Syed, Rahila
Of Michael Paquier Sent: Wednesday, November 26, 2014 1:55 PM To: Alvaro Herrera Cc: Andres Freund; Robert Haas; Fujii Masao; Rahila Syed; Rahila Syed; PostgreSQL-development Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes So, Here are reworked patches for the whole set

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-05 Thread Syed, Rahila
Hello, /* +* We recheck the actual size even if pglz_compress() report success, +* because it might be satisfied with having saved as little as one byte +* in the compressed data. +*/ + *len = (uint16) compressed_len; + if (*len = orig_len - 1) + return false; +

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-06 Thread Syed, Rahila
for scratch buffer. The attached patch introduces OutOfMem flag which is set on when MemoryContextAllocExtended returns NULL . Thank you, Rahila Syed -Original Message- From: Michael Paquier [mailto:michael.paqu...@gmail.com] Sent: Friday, February 06, 2015 12:46 AM To: Syed, Rahila

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-12 Thread Syed, Rahila
Thank you for comments. Please find attached the updated patch. This patch fails to compile: xlogreader.c:1049:46: error: extraneous ')' after condition, expected a statement blk-with_hole blk-hole_offset = 0)) This has been rectified. Note as well

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-18 Thread Syed, Rahila
will post a patch based on this. Thank you, Rahila Syed -Original Message- From: Andres Freund [mailto:and...@2ndquadrant.com] Sent: Monday, February 16, 2015 5:26 PM To: Syed, Rahila Cc: Michael Paquier; Fujii Masao; PostgreSQL mailing lists Subject: Re: [HACKERS] [REVIEW] Re: Compression

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-09 Thread Syed, Rahila
. Thank you, Rahila Syed -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Syed, Rahila Sent: Monday, February 09, 2015 6:58 PM To: Michael Paquier; Fujii Masao Cc: PostgreSQL mailing lists Subject: Re: [HACKERS] [REVIEW] Re

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-09 Thread Syed, Rahila
attached. Thank you, Rahila Syed -Original Message- From: Michael Paquier [mailto:michael.paqu...@gmail.com] Sent: Friday, February 06, 2015 6:00 PM To: Fujii Masao Cc: Syed, Rahila; PostgreSQL mailing lists Subject: Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes On Fri, Feb 6

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-16 Thread Syed, Rahila
Hello, Thank you for reviewing and testing the patch. + /* leave if data cannot be compressed */ + if (compressed_len == 0) + return false; This should be 0, pglz_compress returns -1 when compression fails. + if (pglz_decompress(block_image,

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-03-03 Thread Syed, Rahila
Hello, It would be good to get those problems fixed first. Could you send an updated patch? Please find attached updated patch with WAL replay error fixed. The patch follows chunk ID approach of xlog format. Following are brief measurement numbers.

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-03-04 Thread Syed, Rahila
Hello, Are there any other flag bits that we should or are planning to add into WAL header newly, except the above two? If yes and they are required by even a block which doesn't have an image, I will change my mind and agree to add something like chunk ID to a block header. But I guess the

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-03-05 Thread Syed, Rahila
Hello, Please find attached a patch. As discussed, flag to denote compression and presence of hole in block image has been added in XLogRecordImageHeader rather than block header. Following are WAL numbers based on attached test script posted by Michael earlier in the thread.

Re: [HACKERS] [Proposal] More Vacuum Statistics

2015-06-16 Thread Syed, Rahila
Hello, Maybe, For DBAs, It might be better to show vacuum progress in pg_stat_activity. (if we'd do, add a free-style column like progress ?) This column might also be able to use for other long time commands like ANALYZE, CREATE/RE INDEX and COPY. To realize this feature, we certainly need to

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-02 Thread Syed, Rahila
Hello, Unless I am missing something, I guess you can still keep the actual code that updates counters outside the core if you adopt an approach that Simon suggests. Yes. The code to extract progress information from VACUUM and storing in shared memory can be outside core even with

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-03 Thread Syed, Rahila
Hello, TBH, I think that designing this as a hook-based solution is adding a whole lot of complexity for no value. The hard parts of the problem are collecting the raw data and making the results visible to users, and both of those require involvement of the core code. Where is the benefit

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-08-13 Thread Syed, Rahila
Hello, Autovacuum knows what % of a table needs to be cleaned - that is how it is triggered. When a vacuum runs we should calculate how many TIDs we will collect and therefore how many trips to the indexes we need for given memory. We can use the VM to find out how many blocks we'll need to

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-08-10 Thread Syed, Rahila
Hello, When we're in Phase2 or 3, don't we need to report the number of total page scanned or percentage of how many table pages scanned, as well? The total heap pages scanned need to be reported with phase 2 or 3. Complete progress report need to have numbers from each phase when applicable.

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-16 Thread Syed, Rahila
Hello, Naming the GUC pgstat* seems a little inconsistent. Sorry, there is a typo in the mail. The GUC name is 'track_activity_progress'. Also, adding the new GUC to src/backend/utils/misc/postgresql.conf.sample might be helpful Yes. I will update. Thank you, Rahila Syed

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-10-29 Thread Syed, Rahila
Hello, Please find attached an updated patch. >Flag isn't reset on error. Corrected in the attached. > + pgstat_reset_activityflag; >Does this actually compile? It does compile but with no effect. It has been corrected. >snprintf()? I don't think you need to keep track of schemaname_len

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-10-20 Thread Syed, Rahila
Hello, >I think that you should add the flag or something which indicates whether this >backend is running VACUUM or not, into PgBackendStatus. >pg_stat_vacuum_progress should display the entries of only backends with that >flag set true. This design means that you need to set the flag to true

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-10-06 Thread Syed, Rahila
Hello Fujii-san, >Here are another review comments Thank you for review. Please find attached an updated patch. > You removed some empty lines, for example, in vacuum.h. >Which seems useless to me. Has been corrected in the attached. >Why did you use an array to store the progress information

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-10-06 Thread Syed, Rahila
Hello, Please check the attached patch as the earlier one had typo in regression test output. >+#define PG_STAT_GET_PROGRESS_COLS30 >Why did you use 30? That has come from N_PROGRESS_PARAM * 3 where N_PROGRESS_PARAM = 10 is the number of progress parameters of each type stored in shared

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-09-11 Thread Syed, Rahila
Hello, Please find attached updated VACUUM progress checker patch. Following have been accomplished in the patch 1. Accounts for index pages count while calculating total progress of VACUUM. 2. Common location for storing progress parameters for any command. Idea is every command which needs

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-09-15 Thread Syed, Rahila
Hello Thom, >Okay, I've just tested this with a newly-loaded table (1,252,973 of jsonb >data), Thanks a lot! >but after it's finished, I end up with this: >json=# select * from pg_stat_vacuum_progress; >-[ RECORD 1 ]---+--- >pid | 5569 >total_pages | 217941

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-09-22 Thread Syed, Rahila
Hello, Please find attached patch with bugs reported by Thom and Sawada-san solved. >* The progress of vacuum by autovacuum seems not to be displayed. The progress is stored in shared variables during autovacuum. I guess the reason they are not visible is that the entries are deleted as soon as