Hi Mats, Thanks for the feedback.
First, I would like to clarify the main point. Do you mean that filters and aggregates would run inside the scan TTS? Or would separate plan nodes pass native batches between them? > … requiring a conversion to PostgreSQL-native columnar format > would cost CPU and potentially memory. I do not propose converting a native batch after the scan. A batch can pass between nodes that understand its batch->ops. If a parent does not understand the format, it can materialize only the selected rows and needed columns as Datums. For example, ColumnarScan and a custom batch-aware Agg node could exchange TimescaleDB batches without conversion. Both nodes would remain visible in the plan. > I think it would be more generic to "push down" the operations (for example, > the filters and aggregates) into the TTS and let the TTS implementor (under > control by the TAM implemetor) apply the operator to the batch. I am worried that this could turn TTS into a hidden executor. If TTS executes aggregates, it may later need to execute sorts and joins too. I think TTS should manage the batch format, lifetime, selection, column access, and Datum materialization. Filters, aggregates, sorts, and joins should remain plan nodes. They may be CustomScan nodes that understand a specific native format. > It would be easier and less intrusive to just modify the "getsomeattr" to > accept a Bitmapset instead of an integer. An optional getcolumnattr looks useful. If it fills non-adjacent columns, we need to record which columns are ready, because tts_nvalid can only describe the first N columns. I still think filter columns and output columns should be listed separately. This lets the scan read output columns only for rows that pass the filters. > * Add a function to advance to the next row, which would then do filtering > and aggregation internally and return the resulting "next" row. If the advance > function returns "false", it calls ExecProcNode to get a new batch. An advance callback also looks useful. It should only move within the current batch. The plan node should fetch the next batch and update executor statistics. Best regards, Denis Smirnov
