Hi all, I'd like to propose two improvements to the TsFile format, aiming to speed up query pruning and reduce the cost of reading file metadata.
1. Add table-level statistics Currently, statistics (min/max, count, start/end time, etc.) are only maintained at the page, chunk and timeseries level, stored in ChunkMetadata / TimeseriesMetadata. There are no statistics at the table or device level — the device entries in the metadata index tree only carry the device ID and an offset. As a result, the query engine cannot decide whether a file contains data for a given table within a time range without descending into the metadata index tree and reading the related TimeseriesMetadata. I propose adding per-table statistics to TsFileMetadata, such as per-table row count, min/max timestamps, and whether a table actually has data in the file. This would let the engine prune a file or skip a table before touching the metadata index tree, which is especially beneficial for wide tables and multi-table scenarios. 2. Relocate the table schema map out of TsFileMetadata TsFileMetadata currently stores Map<String, TableSchema> inline in the metadata block. When file metadata is read, the whole block — including the schema region — is loaded from disk, and every TableSchema is still deserialized even when the caller does not need it. TsFileSequenceReader already provides a cacheTableSchemaMap switch and a deserializeWithoutCacheTableSchemaMap path, but these only skip *retaining* the schemas, not *reading and parsing* them. For a table with many columns, the schema region can be large, adding noticeable I/O and deserialization overhead to file opening for queries that only need chunk/page metadata. I propose storing the table schemas outside the hot path, e.g. in a separate metadata region that is read and deserialized only on demand (such as when getTableSchemaMap() is called), so the common metadata-read path pays no cost for the schemas. Feedback and suggestions are welcome. ------------------------------ 大家好， 我想对 TsFile 格式提出两项改进建议，旨在加快查询裁剪速度并降低读取文件元数据的开销。 1. 增加 table 粒度的统计信息 目前统计信息（min/max、count、开始/结束时间等）只维护在 page、chunk 和 timeseries 粒度，存放在 ChunkMetadata / TimeseriesMetadata 中。table 和 device 粒度没有任何统计信息——元数据索引树中的设备条目只保存 deviceID 和偏移量。因此，查询引擎无法在不深入元数据索引树、读取相关 TimeseriesMetadata 的情况下，判断某个文件在给定时间范围内是否包含某张表的数据。 我建议在 TsFileMetadata 中增加 per-table 统计信息，例如每张表的行数、时间戳最小/最大值，以及该表在文件中是否实际包含数据。这样查询引擎可以在触碰元数据索引树之前直接裁剪文件或跳过某张表，对于宽表和多表场景尤其有价值。 2. 将 table schema map 移出 TsFileMetadata TsFileMetadata 目前将 Map<String, TableSchema> 内联存放在元数据块中。读取文件元数据时，整个元数据块（包含 schema 区域）都会从磁盘加载，并且即使调用方并不需要 schema，每个 TableSchema 仍会被反序列化。 TsFileSequenceReader 已经提供了 cacheTableSchemaMap 开关和 deserializeWithoutCacheTableSchemaMap 路径，但它们只能跳过"保留" schema，无法避免"读取和解析" schema。当一个表的列数很多时，schema 区域会比较大，对于只需要 chunk/page 元数据的查询，打开文件会带来明显的 I/O 和反序列化开销。 我建议将 table schema 存放到热路径之外，例如放到独立的元数据区域，仅在需要时（如调用 getTableSchemaMap() 时）才读取和反序列化，使常规的元数据读取路径完全不需要为 schema 付出成本。 欢迎大家提出反馈和建议。 Best regards, Wenwei Shu
