Hi community, I would like to start a discussion about recording the timestamp unit in TsFile metadata.
TsFile stores timestamps as signed 64-bit integers. However, a standalone TsFile currently does not indicate whether those values are expressed in milliseconds, microseconds, or nanoseconds. IoTDB supplies this information through its cluster-level timestamp_precision configuration, but that context may be lost when a file is copied, archived, exchanged between systems, or loaded into another IoTDB deployment. Based on the latest TsFile and IoTDB code, I propose introducing the following standard file-level property: Key: tsfile.timestamp.unit Values: ms, us, or ns Scope: all primary time axes in one TsFile A missing property represents a legacy file whose timestamp unit is unknown This proposal does not change the physical encoding of timestamp columns. The property can be built on the existing TsFile property mechanism, which is already available in the Java, C++, and Python implementations. The proposed TsFile behavior is: Distinguish known, absent, invalid, and unsupported property values. Provide typed reader and writer APIs while retaining the existing generic property APIs. Require the unit to be set before the first timestamp is written and keep it immutable afterward. Validate the reserved property even when it is written through a generic property API. Preserve or validate the unit during file rewriting, splitting, merging, and recovery. Avoid a file-version bump only if the cross-version and cross-language compatibility matrix passes. For IoTDB, the proposed integration is: Write the configured timestamp_precision into every newly created TsFile. Compare the file unit with the target cluster unit during LOAD TSFILE. Allow source-time-unit=ms|us|ns to describe legacy files explicitly. Support timestamp-unit-check=compatible|strict, with compatible as the initial default. Cross-check the file property with the existing timestamp-precision handshake in Pipe. Propagate and validate the property in compaction, WAL recovery, repair, split, and other rewrite paths. Reject mismatched units in the first phase instead of silently relabeling or converting timestamps. Automatic unit conversion is not part of the initial proposal. A correct conversion would need to rewrite timestamps, statistics, indexes, and other time-derived metadata, with explicit overflow, remainder, and rounding rules. The main questions I would like to discuss are: Is tsfile.timestamp.unit an appropriate standard key? Should seconds be supported, or should the initial values remain ms, us, and ns? Should legacy files be accepted by default or require an explicit source unit? Should generic property APIs be allowed to write this reserved key? Is compatibility testing sufficient to avoid a TsFile version bump? Should epoch and other timestamp semantics remain separate from the unit property? The detailed investigation and design draft is available here: https://apache-iotdb-project.feishu.cn/docx/WOIodZzYXo3pcUxv5ZRcIsZUnid Feedback is welcome, especially on the metadata contract and the compatibility strategy for legacy files. Best regards, Hongyin Zhang
