ColinLeeo opened a new issue, #1056:
URL: https://github.com/apache/tsfile/issues/1056

   ## Motivation
   
   Some data sources legitimately produce two points with the same timestamp, 
or deliver points whose order is only non-decreasing. Today both 
implementations reject `time == last_time`, which forces upstream deduplication 
or data loss.
   
   ## Current behavior (verified)
   
   **Java**
   
   - `TabletWriteContext.checkTime` rejects `time <= lastTime`.
   - `NonAlignedChunkGroupWriterImpl.checkIsHistoryData` rejects `time <= 
lastTime`.
   - `AlignedChunkGroupWriterImpl.checkIsHistoryData` rejects `time <= 
lastTime`.
   - `write/page/TimePageWriter` itself performs no ordering check.
   
   **C++**
   
   - `TimePageWriter::write` rejects `timestamp <= statistic_->end_time_`.
   - `TimePageWriter::write_batch` rejects `timestamps[0] <= end_time_` and any 
`timestamps[i] <= timestamps[i-1]`.
   - `TsFileWriter` additionally enforces recovered last-time ordering when 
`enforce_recovered_last_time_order_` is set (append/recovery).
   
   All of these return `E_OUT_OF_ORDER` or throw the out-of-order 
`WriteProcessException`.
   
   Time encoders derive deltas from the input and do not require strictly 
positive deltas: TS_2DIFF stores raw deltas rebased by the minimum delta, so a 
delta of 0 is representable, and Gorilla's delta-of-delta of 0 is likewise 
representable. Still, every time encoder and the read path must be audited 
rather than assumed.
   
   ## Proposal
   
   - Add an **opt-in config flag** (off by default), for example 
`allowNonDecreasingTimestamps` in `TSFileConfig`, a matching `ConfigValue` in 
C++, a C API setter, and a Python keyword argument. When enabled: accept `time 
>= lastTime`; still reject `time < lastTime` with the existing error.
   - Thread the flag through: Java chunk-group writers and tablet contexts, C++ 
`TimePageWriter` single and batch paths, and the recovered-last-time checks in 
`TsFileWriter`.
   - **Audit read behavior and add tests**: statistics updates with equal end 
times, page and chunk readers, aligned chunk reader, time filters and the time 
generator, `BatchData`, `TimeRange`, index entries, and dataset index. Confirm 
that duplicates are returned as separate rows, and document the semantics (no 
deduplication at write time).
   - **Decide and document behavior** for duplicates at page boundaries, chunk 
boundaries, and chunk-group boundaries; through the row API, tablet API, and 
table API; aligned vs non-aligned; and append/restore after recovery.
   - **Tests**: both languages, flag on and off; unit tests plus cross-language 
tests (write duplicates in one language, read in the other).
   
   ## Open questions
   
   - Should duplicates be preserved as separate rows on read, or should the 
writer keep only the last value at a timestamp? Preserving both is the less 
surprising behavior for a storage format; deduplication belongs to the query 
layer.
   - Should the flag also allow out-of-order timestamps within a *tablet batch* 
(`timestamps[i] < timestamps[i-1]`), or only equality?
   
   ## Acceptance criteria
   
   - With the flag off, behavior is unchanged and existing error messages and 
tests still pass.
   - With the flag on, equal timestamps are accepted and read back in order; 
strictly decreasing timestamps still fail.
   - Duplicate semantics are documented and covered across row/tablet/table 
APIs, aligned/non-aligned, and page/chunk/chunk-group boundaries.
   - Cross-language round-trip test (one language writes duplicates, the other 
reads them).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to