Hello All, Recently, I have been involved in bringing derived columns(aka generated columns) for the Iceberg connector in Presto Link: https://github.com/prestodb/rfcs/pull/61. We currently use table properties to store the extra metadata for derived columns. However, we cannot achieve the following:
1. Cross engine compatibility: How do we know if the derived columns metadata is in sync i.e. if other engines unaware of derived columns write to the table/alter the columns etc... 2. UDF portability and versioning: which is already addressed by (Iceberg UDF spec) 3. VIRTUAL columns: We store all the column metadata/metrics/stats including a bloom filter but not the actual data. A poor man's index. We are considering a Iceberg spec level support to address all of the above. Just wondering, is there work already happening in this direction elsewhere? If not, I have started an issue: https://github.com/apache/iceberg/issues/15923 , will be adding more details soon. Thanks, Prashant
