Hi Denis, Erik, I agree with you that we do not need to modify the TAM interface and can stay with just modifying the TTS, however, I see some compllications with the suggested approach.
The first thing is that I think there is no need to introduce a new data structure for dealing with columnar selection. It would be easier and less intrusive to just modify the "getsomeattr" to accept a Bitmapset instead of an integer. It is an ABI change, but there we are not yet at a stage with PostgreSQL where you can actually keep an ABI anyway. (If we for some reason need to maintain the ABI, we can just add a method, e.g., "getcolumnattr" to the end and use that instead of "getsomeattr" if it is none-null. This also naturally falls back to the old implementation and allows, e.g., heapam to not require modifications.) After calling this method, only the mentioned columns will be populated, allowing a TAM to either honor the bitmap precisely, or fetch more columns than necessary if it has a row format (it won't break anything, but it is unnecessary). For the TTS implementation that we did for Hypercore, we passed back a batch and we had to add custom scan nodes to read the rows of the batch one by one (correct me if I remember incorrectly here Erik). This we could do because we already had the rows in columnar format (more or less), which made the transformation cheap. On Sat, Aug 15, 2026 at 11:08 AM Denis Smirnov <[email protected]> wrote: > Hi Erik, > > Thank you. Your slot-based design is very close to the direction I think we > should take. > > The idea is to generalize tts_batch for both heap and columnar storage. We > do not want to add a new table AM callback. A batch should keep its native > data for as long as possible. For heap, this would be an array of Datum > values for each column, built from a pinned buffer page. A columnar slot > could keep Arrow arrays, compressed columns, dictionaries, or any other > extension-specific data. Conversion to Datum would happen only when a > regular PostgreSQL node asks for rows. > But the format of a batch might be different depending on the storage engine, so requiring a conversion to PostgreSQL-native columnar format would cost CPU and potentially memory. Since the end goal for this is to be able to apply an operation to a batch of rows, I think it would be more generic to "push down" the operations (for example, the filters and aggregates) into the TTS and let the TTS implementor (under control by the TAM implemetor) apply the operator to the batch. That way, the TAM implementor can keep whatever format they have decided is best for the implementation, but in turn need to apply the operation themselves. If we want to support some native format, such as Arrow, we can add a library with operations that make this easy to implement. So, roughly these changes to the TTS: * Allow adding a filter to a slot through a function. If it is NULL, revert to old behaviour. * Allow pushing down an aggregate function into a batch. * Add a function to advance to the next row, which would then do filtering and aggregation internally and return the resulting "next" row. If the advance function returns "false", it calls ExecProcNode to get a new batch. It would require changes to the scan nodes as well, roughly this if we just consider SeqScan: 1. Call ExecProcNode to get a new "batch" (it could be a single row, as before). 2. If the push-down function exists, push down the filter. 3. call getsomeattr to get the attributes from the tuple 4. Pass it upwards. 5. When a request for the next tuple comes, check if "advance" exists and call it in that case. 6. If advance exists and returns true, go back to step 3. > 6. If advance did not exist, or it returned false. Go back to step 1. WDYT? > -- > Best wishes, Mats Kindahl, Multigres Developer, Supabase
