Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.
Hello, Yes. Any percent completion calculation will have to account for the case of needing multiple passes through all the indexes. Each dead tuple requires 6 bytes (IIRC) of maintenance work mem. So if you're deleting 5M rows with m_w_m=1MB you should be getting many passes through the indexes. Studying the output of VACUUM VERBOSE will confirm that (or just throw a temporary WARNING in the path where we start the scan). Yes I see the problem now. I get the message WARNING: Overall index percentage completion 100.00 logged 25 times while vacuuming after 5 million records deleted. Figuring out number of multiple index passes beforehand, accurately, is the problem to solve. Clearly need to study this some more. Thank you, Sameer Thakur | Senior Software Specialist | NTTDATA Global Delivery Services Private Ltd | w. +91.20.6641.7146 | VoIP: 8834.8146 | m. +91 989.016.6656 | sameer.tha...@nttdata.com | Follow us on Twitter@NTTDATAAmericas __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.
Hello, logged 25 times Sorry, it is much lower at 7 times. Does not change overall point though regards Sameer Thakur | Senior Software Specialist | NTTDATA Global Delivery Services Private Ltd | w. +91.20.6641.7146 | VoIP: 8834.8146 | m. +91 989.016.6656 | sameer.tha...@nttdata.com | Follow us on Twitter@NTTDATAAmericas __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.
Hello, I think it'd be better to combine both numbers into one report: It'd also be good to standardize on where the * 100 is happening. Done can be replaced by (itemptr-ipblkid != vacrelstats-last_scanned_page) Get compiler error : invalid operands to binary != (have ‘BlockIdData’ and ‘BlockIdData’) vacrelstats-current_index_scanned_page_count++; Done Please find v3 attached. I am struggling to create maintenance work memory exhaustion. Did the following maintenance_work_mem=1MB. Inserted 10 million records in tbl1 with 3 indexes. Deleted 5 million and vacuumed. So far no error. I could keep bumping up the records to say 100 million and try to get this error. This seems a tedious manner to simulate maintenance work memory exhaustion. Is there a better way? To insert I am using COPY (from a csv which has 10 million records) and building indexes after insert is complete. Thank you Sameer __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. IndexScanProgress_v3.patch Description: IndexScanProgress_v3.patch -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.
Hello, Does this actually handle multiple indexes? It doesn't appear so, which I'd think is a significant problem... :/ Please find v2 attached which does this. I'm also not seeing how this will deal with exhausting maintenance_work_mem. ISTM that when that happens you'd definitely want a better idea of what's going on... Will work on this aspect in v3. Thank you, Sameer __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. IndexScanProgress_v2.patch Description: IndexScanProgress_v2.patch -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.
Hello, I am not really willing to show up as the picky guy here, but could it be possible to receive those patches as attached to emails instead of having them referenced by URL? I imagine that you are directly using the nabble interface. Just configured a new mail client for nabble, did not know how to use it within an existing conversation. Now I can send the patch attached! Thanks Sameer __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. IndexScanProgress.patch Description: IndexScanProgress.patch -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
