ColinLeeo opened a new issue, #1056: URL: https://github.com/apache/tsfile/issues/1056
## Motivation Some data sources legitimately produce two points with the same timestamp, or deliver points whose order is only non-decreasing. Today both implementations reject `time == last_time`, which forces upstream deduplication or data loss. ## Current behavior (verified) **Java** - `TabletWriteContext.checkTime` rejects `time <= lastTime`. - `NonAlignedChunkGroupWriterImpl.checkIsHistoryData` rejects `time <= lastTime`. - `AlignedChunkGroupWriterImpl.checkIsHistoryData` rejects `time <= lastTime`. - `write/page/TimePageWriter` itself performs no ordering check. **C++** - `TimePageWriter::write` rejects `timestamp <= statistic_->end_time_`. - `TimePageWriter::write_batch` rejects `timestamps[0] <= end_time_` and any `timestamps[i] <= timestamps[i-1]`. - `TsFileWriter` additionally enforces recovered last-time ordering when `enforce_recovered_last_time_order_` is set (append/recovery). All of these return `E_OUT_OF_ORDER` or throw the out-of-order `WriteProcessException`. Time encoders derive deltas from the input and do not require strictly positive deltas: TS_2DIFF stores raw deltas rebased by the minimum delta, so a delta of 0 is representable, and Gorilla's delta-of-delta of 0 is likewise representable. Still, every time encoder and the read path must be audited rather than assumed. ## Proposal - Add an **opt-in config flag** (off by default), for example `allowNonDecreasingTimestamps` in `TSFileConfig`, a matching `ConfigValue` in C++, a C API setter, and a Python keyword argument. When enabled: accept `time >= lastTime`; still reject `time < lastTime` with the existing error. - Thread the flag through: Java chunk-group writers and tablet contexts, C++ `TimePageWriter` single and batch paths, and the recovered-last-time checks in `TsFileWriter`. - **Audit read behavior and add tests**: statistics updates with equal end times, page and chunk readers, aligned chunk reader, time filters and the time generator, `BatchData`, `TimeRange`, index entries, and dataset index. Confirm that duplicates are returned as separate rows, and document the semantics (no deduplication at write time). - **Decide and document behavior** for duplicates at page boundaries, chunk boundaries, and chunk-group boundaries; through the row API, tablet API, and table API; aligned vs non-aligned; and append/restore after recovery. - **Tests**: both languages, flag on and off; unit tests plus cross-language tests (write duplicates in one language, read in the other). ## Open questions - Should duplicates be preserved as separate rows on read, or should the writer keep only the last value at a timestamp? Preserving both is the less surprising behavior for a storage format; deduplication belongs to the query layer. - Should the flag also allow out-of-order timestamps within a *tablet batch* (`timestamps[i] < timestamps[i-1]`), or only equality? ## Acceptance criteria - With the flag off, behavior is unchanged and existing error messages and tests still pass. - With the flag on, equal timestamps are accepted and read back in order; strictly decreasing timestamps still fail. - Duplicate semantics are documented and covered across row/tablet/table APIs, aligned/non-aligned, and page/chunk/chunk-group boundaries. - Cross-language round-trip test (one language writes duplicates, the other reads them). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
