Hi, I remember that the sync module will rename the file after merge. More details about sync could be given by @Tianan.
Best, -- Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 > -----原始邮件----- > 发件人: "Xiangdong Huang" <[email protected]> > 发送时间: 2019-09-02 13:28:50 (星期一) > 收件人: [email protected] > 抄送: > 主题: Re: [IoTDB-189] question about the feature of compatibility of HDFS > > Hi, > > Look forward to the performance comparison. > > > Currently, renaming only occurs in two case > > (1) updating the version files for each storage group periodically in > system/storage_group_x/version-xxx > > (2) updating the compression-ratio file when dynamically adjusting > parameters > > How about the sync module? Seems there is a file for recording the file > list (well, the sync module execution is not frequent) > > Best, > > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Jialin Qiao <[email protected]> 于2019年9月1日周日 下午8:00写道： > > > Hi, > > > > > Ah... I remember that there is a file whose name is "version-xxx", in > > which > > the "xxx" is a counter, and will be updated if the version difference > > between the disk and memory >= 50. > > > > Yes, each storage group has a file named version-xxx under > > system/storage_group_xxx folder. > > > > > I am not sure whether it is correct that "renaming is not very frequent" > > because each ChunkGroup has a version (am I right?).. > > > > Right, each ChunkGroup will increase this version, and each new data file > > will increase this version. > > > > Currently, renaming only occurs in two case > > (1) updating the version files for each storage group periodically in > > system/storage_group_x/version-xxx > > (2) updating the compression-ratio file when dynamically adjusting > > parameters > > > > It's hard to say whether it is frequent or not, we can compare the > > performance between local file system and HDFS after supporting HDFS. > > > > Best, > > -- > > Jialin Qiao > > School of Software, Tsinghua University > > > > 乔嘉林 > > 清华大学 软件学院 > > > > > -----原始邮件----- > > > 发件人: "Xiangdong Huang" <[email protected]> > > > 发送时间: 2019-09-01 23:10:08 (星期日) > > > 收件人: [email protected] > > > 抄送: > > > 主题: Re: [IoTDB-189] question about the feature of compatibility of HDFS > > > > > > Hi, > > > > > > > since renaming is not very frequent, maybe we do not need to worry > > about > > > performance. > > > > > > Ah... I remember that there is a file whose name is "version-xxx", in > > which > > > the "xxx" is a counter, and will be updated if the version difference > > > between the disk and memory >= 50. > > > > > > I am not sure whether it is correct that "renaming is not very frequent" > > > because each ChunkGroup has a version (am I right?).. > > > > > > But, I am not saying that writing on HDFS is bad, I just want to remind > > you > > > that reducing renaming operations, or proving that renaming on HDFS is > > not > > > a time consuming operation. > > > > > > Best, > > > ----------------------------------- > > > Xiangdong Huang > > > School of Software, Tsinghua University > > > > > > 黄向东 > > > 清华大学 软件学院 > > > > > > > > > Zesong Sun <[email protected]> 于2019年8月31日周六 下午8:28写道： > > > > > > > Hi, > > > > > > > > > > > > This issue is a sub-task of IoTDB-187 [1], which enables users to > > choose > > > > whether files are storied in local file system or HDFS. > > > > > > > > > > > > In this sub-task, I intend to use an encapsulated "File" to replace > > > > existed Java File in our project. > > > > > > > > > > > > Currently in our design, all files could be written on HDFS. > > > > > > > > > > > > HDFS supports truncating files, and since renaming is not very > > frequent, > > > > maybe we do not need to worry about performance. > > > > > > > > > > > > If anyone has any other ideas, please discuss with me. > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IOTDB-187 > > > > > > > > > > > > ------------------ > > > > Zesong Sun > > > > School of Software, Tsinghua University > > > > > > > > 孙泽嵩 > > > > 清华大学 软件学院 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------ 原始邮件 ------------------ > > > > 发件人: "Xiangdong Huang"<[email protected]>; > > > > 发送时间: 2019年8月31日(星期六) 晚上10:50 > > > > 收件人: "dev"<[email protected]>; > > > > > > > > 主题: [IoTDB-189] question about the feature of compatibility of HDFS > > > > > > > > > > > > > > > > Hi, > > > > > > > > This issue only has a title.. without any description... > > > > > > > > I think it is for letting IoTDB write TsFiles on HDFS directly. > > > > > > > > My question is, is only TsFile written on HDFS? or all files (e.g., > > wal, > > > > system data file)? > > > > > > > > Besides, if all files are on HDFS, do we need to avoid using File name > > as a > > > > mark or a lock (i.e., create a file and then rename it) for performance > > > > consideration? > > > > > > > > (I am on my trip and then I realize clearly that current info on issue > > is > > > > not enough for guys who are on the internet) > > > > > > > > Best, > > > > ----------------------------------- > > > > Xiangdong Huang > > > > School of Software, Tsinghua University > > > > > > > > 黄向东 > > > > 清华大学 软件学院 > >
