Hi community,

I would like to start a discussion about recording the timestamp unit in TsFile 
metadata.

TsFile stores timestamps as signed 64-bit integers. However, a standalone 
TsFile currently does not indicate whether those values are expressed in 
milliseconds, microseconds, or nanoseconds. IoTDB supplies this information 
through its cluster-level timestamp_precision configuration, but that context 
may be lost when a file is copied, archived, exchanged between systems, or 
loaded into another IoTDB deployment.

Based on the latest TsFile and IoTDB code, I propose introducing the following 
standard file-level property:

Key: tsfile.timestamp.unit

Values: ms, us, or ns

Scope: all primary time axes in one TsFile

A missing property represents a legacy file whose timestamp unit is unknown

This proposal does not change the physical encoding of timestamp columns. The 
property can be built on the existing TsFile property mechanism, which is 
already available in the Java, C++, and Python implementations.

The proposed TsFile behavior is:

Distinguish known, absent, invalid, and unsupported property values.

Provide typed reader and writer APIs while retaining the existing generic 
property APIs.

Require the unit to be set before the first timestamp is written and keep it 
immutable afterward.

Validate the reserved property even when it is written through a generic 
property API.

Preserve or validate the unit during file rewriting, splitting, merging, and 
recovery.

Avoid a file-version bump only if the cross-version and cross-language 
compatibility matrix passes.

For IoTDB, the proposed integration is:

Write the configured timestamp_precision into every newly created TsFile.

Compare the file unit with the target cluster unit during LOAD TSFILE.

Allow source-time-unit=ms|us|ns to describe legacy files explicitly.

Support timestamp-unit-check=compatible|strict, with compatible as the initial 
default.

Cross-check the file property with the existing timestamp-precision handshake 
in Pipe.

Propagate and validate the property in compaction, WAL recovery, repair, split, 
and other rewrite paths.

Reject mismatched units in the first phase instead of silently relabeling or 
converting timestamps.

Automatic unit conversion is not part of the initial proposal. A correct 
conversion would need to rewrite timestamps, statistics, indexes, and other 
time-derived metadata, with explicit overflow, remainder, and rounding rules.

The main questions I would like to discuss are:

Is tsfile.timestamp.unit an appropriate standard key?

Should seconds be supported, or should the initial values remain ms, us, and ns?

Should legacy files be accepted by default or require an explicit source unit?

Should generic property APIs be allowed to write this reserved key?

Is compatibility testing sufficient to avoid a TsFile version bump?

Should epoch and other timestamp semantics remain separate from the unit 
property?

The detailed investigation and design draft is available here:

https://apache-iotdb-project.feishu.cn/docx/WOIodZzYXo3pcUxv5ZRcIsZUnid

Feedback is welcome, especially on the metadata contract and the compatibility 
strategy for legacy files.

Best regards,
Hongyin Zhang

Reply via email to