On Tue, Nov 17, 2020 at 6:56 PM Etsuro Fujita <etsuro.fuj...@gmail.com> wrote: > * I haven't yet done anything about the issue on postgres_fdw's > handling of concurrent data fetches by multiple ForeignScan nodes > (below *different* Append nodes in the query) using the same > connection discussed in [2]. I modified the patch to just disable > applying this feature to problematic test cases in the postgres_fdw > regression tests, by a new GUC enable_async_append.
A solution for the issue would be a scheduler designed to handle such data fetches more efficiently, but I don’t think it’s easy to create such a scheduler. Rather than doing so, I'd like to propose to allow FDWs to disable async execution of them in problematic cases by themselves during executor startup in the first cut. What I have in mind for that is: 1) For an FDW that has async-capable ForeignScan(s), we allow the FDW to record, for each of the async-capable and non-async-capable ForeignScan(s), the information on a connection to be used for the ForeignScan into EState during BeginForeignScan(). 2) After doing ExecProcNode() to each SubPlan and the main query tree in InitPlan(), we give the FDW a chance to a) reconsider, for each of the async-capable ForeignScan(s), whether the ForeignScan can be executed asynchronously as planned, based on the information stored into EState in #1, and then b) disable async execution of the ForeignScan if not. #1 and #2 would be done after initial partition pruning, so more async-capable ForeignScans would be executed asynchronously, if other async-capable ForeignScans conflicting with them are removed by that pruning. This wouldn’t prevent us from adding a feature like what was proposed by Horiguchi-san later. BTW: while considering this, I noticed some bugs with ExecAppendAsyncBegin() in the previous patch. Attached is a new version of the patch fixing them. I also tweaked some comments a little bit. Best regards, Etsuro Fujita
async-wip-2021-02-01.patch
Description: Binary data
