Hi, Look forward to the performance comparison.
> Currently, renaming only occurs in two case > (1) updating the version files for each storage group periodically in system/storage_group_x/version-xxx > (2) updating the compression-ratio file when dynamically adjusting parameters How about the sync module? Seems there is a file for recording the file list (well, the sync module execution is not frequent) Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao <[email protected]> 于2019年9月1日周日 下午8:00写道： > Hi, > > > Ah... I remember that there is a file whose name is "version-xxx", in > which > the "xxx" is a counter, and will be updated if the version difference > between the disk and memory >= 50. > > Yes, each storage group has a file named version-xxx under > system/storage_group_xxx folder. > > > I am not sure whether it is correct that "renaming is not very frequent" > because each ChunkGroup has a version (am I right?).. > > Right, each ChunkGroup will increase this version, and each new data file > will increase this version. > > Currently, renaming only occurs in two case > (1) updating the version files for each storage group periodically in > system/storage_group_x/version-xxx > (2) updating the compression-ratio file when dynamically adjusting > parameters > > It's hard to say whether it is frequent or not, we can compare the > performance between local file system and HDFS after supporting HDFS. > > Best, > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > -----原始邮件----- > > 发件人: "Xiangdong Huang" <[email protected]> > > 发送时间: 2019-09-01 23:10:08 (星期日) > > 收件人: [email protected] > > 抄送: > > 主题: Re: [IoTDB-189] question about the feature of compatibility of HDFS > > > > Hi, > > > > > since renaming is not very frequent, maybe we do not need to worry > about > > performance. > > > > Ah... I remember that there is a file whose name is "version-xxx", in > which > > the "xxx" is a counter, and will be updated if the version difference > > between the disk and memory >= 50. > > > > I am not sure whether it is correct that "renaming is not very frequent" > > because each ChunkGroup has a version (am I right?).. > > > > But, I am not saying that writing on HDFS is bad, I just want to remind > you > > that reducing renaming operations, or proving that renaming on HDFS is > not > > a time consuming operation. > > > > Best, > > ----------------------------------- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > 黄向东 > > 清华大学 软件学院 > > > > > > Zesong Sun <[email protected]> 于2019年8月31日周六 下午8:28写道： > > > > > Hi, > > > > > > > > > This issue is a sub-task of IoTDB-187 [1], which enables users to > choose > > > whether files are storied in local file system or HDFS. > > > > > > > > > In this sub-task, I intend to use an encapsulated "File" to replace > > > existed Java File in our project. > > > > > > > > > Currently in our design, all files could be written on HDFS. > > > > > > > > > HDFS supports truncating files, and since renaming is not very > frequent, > > > maybe we do not need to worry about performance. > > > > > > > > > If anyone has any other ideas, please discuss with me. > > > > > > > > > [1] https://issues.apache.org/jira/browse/IOTDB-187 > > > > > > > > > ------------------ > > > Zesong Sun > > > School of Software, Tsinghua University > > > > > > 孙泽嵩 > > > 清华大学 软件学院 > > > > > > > > > > > > > > > > > > > > > > > > ------------------ 原始邮件 ------------------ > > > 发件人: "Xiangdong Huang"<[email protected]>; > > > 发送时间: 2019年8月31日(星期六) 晚上10:50 > > > 收件人: "dev"<[email protected]>; > > > > > > 主题: [IoTDB-189] question about the feature of compatibility of HDFS > > > > > > > > > > > > Hi, > > > > > > This issue only has a title.. without any description... > > > > > > I think it is for letting IoTDB write TsFiles on HDFS directly. > > > > > > My question is, is only TsFile written on HDFS? or all files (e.g., > wal, > > > system data file)? > > > > > > Besides, if all files are on HDFS, do we need to avoid using File name > as a > > > mark or a lock (i.e., create a file and then rename it) for performance > > > consideration? > > > > > > (I am on my trip and then I realize clearly that current info on issue > is > > > not enough for guys who are on the internet) > > > > > > Best, > > > ----------------------------------- > > > Xiangdong Huang > > > School of Software, Tsinghua University > > > > > > 黄向东 > > > 清华大学 软件学院 >
