[jira] [Commented] (IOTDB-352) [Distributed] Recognize and skip duplicated files in a snapshot
[ https://issues.apache.org/jira/browse/IOTDB-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005984#comment-17005984 ] Tian Jiang commented on IOTDB-352: -- This is the current solution, it is not perfect. Any suggestions or new ideas are welcomed. > [Distributed] Recognize and skip duplicated files in a snapshot > --- > > Key: IOTDB-352 > URL: https://issues.apache.org/jira/browse/IOTDB-352 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Tian Jiang >Priority: Major > Labels: distributed > > By the naming of TsFiles in IoTDB, the files with the same data may have > different names on different nodes. When such files are sent through > snapshots, the receiver is unable to tell whether the file already exists > locally or not, so it will blindly load the file as an unsequential one (if > it does overlap any existing files), which will waste a lot of system > resources. > How can we figure out if we already have one file or not? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-352) [Distributed] Recognize and skip duplicated files in a snapshot
[ https://issues.apache.org/jira/browse/IOTDB-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005983#comment-17005983 ] Tian Jiang commented on IOTDB-352: -- Adding md5 is not helpful in this issue, it may be used to check the file integrity of files during file transfers, but that is another issue. > [Distributed] Recognize and skip duplicated files in a snapshot > --- > > Key: IOTDB-352 > URL: https://issues.apache.org/jira/browse/IOTDB-352 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Tian Jiang >Priority: Major > Labels: distributed > > By the naming of TsFiles in IoTDB, the files with the same data may have > different names on different nodes. When such files are sent through > snapshots, the receiver is unable to tell whether the file already exists > locally or not, so it will blindly load the file as an unsequential one (if > it does overlap any existing files), which will waste a lot of system > resources. > How can we figure out if we already have one file or not? -- This message was sent by Atlassian Jira (v8.3.4#803005)
