[
https://issues.apache.org/jira/browse/IOTDB-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275523#comment-17275523
]
Chao Wang commented on IOTDB-1140:
----------------------------------
Thanks， I know your mean..
I have some idea，we can think about it.
# Revise to correct data, which better reflects the benefits of this algorithm.
# This is what users want. They occasionally insert an incorrect line of data,
which is not their intention, and the algorithm can give a corrent value， not
delete it..
# when page write , The time-column is encoded first, and then the
value-column is encoded. When find the value is incorrect, it is difficult to
process remove timestamp.
> optimize regular data encoding
> ------------------------------
>
> Key: IOTDB-1140
> URL: https://issues.apache.org/jira/browse/IOTDB-1140
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/Engine
> Reporter: Chao Wang
> Assignee: Chao Wang
> Priority: Critical
>
> current regular data encoding algorithm：
> # Calculate the difference between two adjacent values. The smallest
> difference is used as the equal-frequency frequency.
> # Determine the data range of this batch of data based on the difference
> between the last value and the first value.
> # Traverse this batch of data, use a BitSet, compare the difference between
> two adjacent values with the same frequency, and save the value true by
> default,
> If the value is not equal to the equal frequency, calculate the number of
> equal frequency differences and set the value to false at the corresponding
> position, indicating that the point is a missing point.
>
> this algorithm only can identity missing point, if have error point , it
> will throw exception..
> because BitSet only can do this thing, indicates whether the same frequency
> exists in a segment of data
>
> But there is some optimize point..
> If there is an abnormal value in a column of values, the algorithm is
> deviated if the difference is directly obtained to the minimum value.
> sample: 1000，1100，1800，1400，1500...
> current algorithm be do not use...
> 1800 is a error point, we should identity error point, revise data.
> revise data should be : 1000，1100，1300，1400，1500
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
