On Sun, Feb 19, 2023 at 21:06 PM Wang, Wei/王 威 <wangw.f...@fujitsu.com> wrote: > On Thur, Feb 14, 2023 at 2:03 AM Andres Freund <and...@anarazel.de> wrote: > > On 2023-02-13 14:06:57 +0530, Amit Kapila wrote: > > > > > The patch calls update_progress in change_cb_wrapper and other > > > > > wrappers which will miss the case of DDLs that generates a lot of data > > > > > that is not processed by the plugin. I think for that we either need > > > > > to call update_progress from reorderbuffer.c similar to what the patch > > > > > has removed or we need some other way to address it. Do you have any > > > > > better idea? > > > > > > > > I don't mind calling something like update_progress() in the specific > > > > cases > > > > that's needed, but I think those are just the > > > > if (!RelationIsLogicallyLogged(relation)) > > > > if (relation->rd_rel->relrewrite && !rb->output_rewrites)) > > > > > > > > To me it makes a lot more sense to call update_progress() for those, > > > > rather > > > > than generally. > > > > > > > > > > Won't it be better to call it wherever we don't invoke any wrapper > > > function like for cases REORDER_BUFFER_CHANGE_INVALIDATION, sequence > > > changes, etc.? I was thinking that wherever we don't call the wrapper > > > function which means we don't have a chance to invoke > > > update_progress(), the timeout can happen if there are a lot of such > > > messages. > > > > ISTM that the likelihood of causing harm due to increased overhead is higher > > than the gain. > > I would like to do something for this thread. So, I am planning to update the > patch as per discussion in the email chain unless someone is already working > on > it.
Thanks to Andres and Amit for the discussion. Based on the discussion and Andres' WIP(in [1]), I made the following modifications: 1. Some function renaming stuffs. 2. Added the threshold-related logic in the function update_progress_and_keepalive. 3. Added the timeout-related processing of temporary data and unlogged/foreign/system tables in the function ReorderBufferProcessTXN. 4. Improved error messages in the function OutputPluginPrepareWrite. 5. Invoked function update_progress_and_keepalive to fix sync-related problems caused by filters such as origin in functions DecodeCommit(), DecodePrepare() and ReorderBufferAbort(); 6. Removed the invocation of function update_progress_and_keepalive in the function begin_prepare_cb_wrapper(). 7. Invoked the function update_progress_and_keepalive() in the function stream_truncate_cb_wrapper(), just like we do in the function truncate_cb_wrapper(). 8. Removed the check of SyncRepRequested() in the syncrep logic in the function WalSndUpdateProgressAndKeepAlive(); 9. Added the check for wal_sender_timeout before using it in functions WalSndUpdateProgressAndKeepAlive() and WalSndWriteData(); Attach the new patch. [1] - https://www.postgresql.org/message-id/20230208200235.esfoggsmuvf4pugt%40awork3.anarazel.de Regards, Wang wei
v2-0001-Rework-LogicalOutputPluginWriterUpdateProgress.patch
Description: v2-0001-Rework-LogicalOutputPluginWriterUpdateProgress.patch