Hello, A few years back[1] I experimented with a simple readiness API that would allow Append to start emitting tuples from whichever Foreign Scan has data available, when working with FDW-based sharding. I used that primarily as a way to test Andres's new WaitEventSet stuff and my kqueue implementation of that, but I didn't pursue it seriously because I knew we wanted a more ambitious async executor rewrite and many people had ideas about that, with schedulers capable of jumping all over the tree etc.
Anyway, Stephen Frost pinged me off-list to ask about that patch, and asked why we don't just do this naive thing until we have something better. It's a very localised feature that works only between Append and its immediate children. The patch makes it work for postgres_fdw, but it should work for any FDW that can get its hands on a socket. Here's a quick rebase of that old POC patch, along with a demo. Since 2016, Parallel Append landed, but I didn't have time to think about how to integrate with that so I did a quick "sledgehammer" rebase that disables itself if parallelism is in the picture. === demo === create table t (a text, b text); create or replace function slow_data(name text) returns setof t as $$ begin perform pg_sleep(random()); return query select name, generate_series(1, 100)::text as i; end; $$ language plpgsql; create view t1 as select * from slow_data('t1'); create view t2 as select * from slow_data('t2'); create view t3 as select * from slow_data('t3'); create extension postgres_fdw; create server server1 foreign data wrapper postgres_fdw options (dbname 'postgres'); create server server2 foreign data wrapper postgres_fdw options (dbname 'postgres'); create server server3 foreign data wrapper postgres_fdw options (dbname 'postgres'); create user mapping for current_user server server1; create user mapping for current_user server server2; create user mapping for current_user server server3; create foreign table ft1 (a text, b text) server server1 options (table_name 't1'); create foreign table ft2 (a text, b text) server server2 options (table_name 't2'); create foreign table ft3 (a text, b text) server server3 options (table_name 't3'); -- create three remote shards create table pt (a text, b text) partition by list (a); alter table pt attach partition ft1 for values in ('ft1'); alter table pt attach partition ft2 for values in ('ft2'); alter table pt attach partition ft3 for values in ('ft3'); -- see that tuples come back in the order that they're ready 
select * from pt where b like '42'; [1] https://www.postgresql.org/message-id/CAEepm%3D1CuAWfxDk%3D%3DjZ7pgCDCv52fiUnDSpUvmznmVmRKU5zpA%40mail.gmail.com -- Thomas Munro https://enterprisedb.com
0001-Multiplexing-Append-POC.patch
Description: Binary data