Hi Xiangdong, Great work! I'll try to go through your code to understand the internals better and perhaps do a simulation test with hard jvm exits. If everything goes fine we will incorporate that asap in test code and give you feedback!
Thank you! Julian Von meinem Mobiltelefon gesendet -------- Ursprüngliche Nachricht -------- Betreff: Re: Operation and robustness of iotDB Von: Xiangdong Huang An: dev@iotdb.apache.org Cc: Hi, I have added a new TsFileIOWriter, which supports to recover data from a broken TsFile (I mean, a TsFile that is not closed correctly). The PR is https://github.com/apache/incubator-iotdb/pull/87. You can use the new feature by: ``` NativeRestorableIOWriter rWriter = new NativeRestorableIOWriter(file); TsFileWriter writer = new TsFileWriter(rWriter); ``` 1. If the file is a complete TsFile (e.g., the FileMetadata has been persistent into the file and the tail magic string is correct), then you can not write data into this TsFile anymore (actually I can remove all FileMetadata and then let you continue to write, but it needs more coding work...), `TsFileWriter writer = new TsFileWriter(rWriter);` will throw a IoException. You can check it before you create TsFileWriter by using `rWriter.canWrite()`. 2. ChunkGroup is the basic unit that can be recover. That is, if a ChunkGroup is complete, i.e., its chunkGroupFooter is complete, I will remain it, otherwise I will truncate the chunk group. However, I can not know whether a chunkGroupFooter is complete (if only one byte lose, the deserialize method does not throw exception, because something like `deserializeToInt(inputstream)` does not throw exception (it will use 3 bytes to convert them to a integer). So, when I say "if a ChunkGroup is complete", I mean that not only its chunkGroupFooter is serialized successfully, but also there is at least one more byte is serialized into the file followed the footer. 3. I have tested many cases, such only a magic head string is persisted, only a ChunkHeader is persisted, the first ChunkHeader is not persisted completely, only some Chunks are persisted, a ChunkGroupFooter is persisted, Two ChunkGroupFooter are persisted, and a complete TsFile is persisted, etc.. (`NativeRestorableIOWriterTest.java`, hope I have cover all cases) Hope TsFile can be used in edge device management applications ASAP! Now please enjoy the new feature. ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Julian Feinauer <j.feina...@pragmaticminds.de> 于2019年3月5日周二 下午8:10写道： > Hey, > > thank you fort he link... I did not know of this.. this is exactly what I > was looking for! > > Julian > > PS.: Looking forward to your PR : ) > > Am 05.03.19, 12:26 schrieb "Xiangdong Huang" <saint...@gmail.com>: > > Hi, > > 1. We have a document to introduce that: > https://cwiki.apache.org/confluence/display/IOTDB/TsFile+Format > > 2. The new API for recovering data is almost done. I am writing the UTs > now. Maybe I can submit a PR tonight (if everything is fine...) > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Julian Feinauer <j.feina...@pragmaticminds.de> 于2019年3月5日周二 下午6:00写道： > > > Hi Xiangdong, > > > > that sounds excellent. > > Do you have a short overview of how the file format is designed on > disk? > > I know that its somewhat similar to parquet but I did not find more > > details. > > Basically what would suffice for us would be something like skipping > an > > invalid column group (or how you name it) and go on with the next, > or so. > > > > Julian > > > > Am 04.03.19, 13:21 schrieb "Xiangdong Huang" <saint...@gmail.com>: > > > > Hi, > > > > If so, I think I need to add a new API to allow you continue to > write > > data > > in an existing but not closed correctly TsFile. Then everything > is > > fine > > for you :D > > > > Best, > > ----------------------------------- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > 黄向东 > > 清华大学 软件学院 > > > > > > Julian Feinauer <j.feina...@pragmaticminds.de> 于2019年3月4日周一 > 下午8:08写道： > > > > > Hey Xiangdong, > > > > > > thanks for the great explanation. > > > And in fact, I agree with you that it would be best if we > start to > > play > > > around with it and reply all our findings or wishes back to > this > > list (in > > > fact that proved to be beneficial in plc4x as well). > > > > > > You confirm my thoughts about the two "levels" of APIs (DB and > file) > > and > > > the file api is exactly what we looked for for our use case. > > > As we do not care much about data loss (when an edge device > fails > > its... > > > gone). > > > The crucial point for us is that no corrupt files can be > generated. > > > This means I'm fine when the last data submitted is lost but > I'm not > > fine > > > if we can get to a situation where the last datafile is > completely > > lost > > > (well, perhaps this could be acceptable). > > > > > > @tim: Perhaps its best when you give some more information to > > Xiangdong > > > about our idea, and we can also point to our current code in > github > > > > > > Julian > > > > > > Am 04.03.19, 13:03 schrieb "Xiangdong Huang" < > saint...@gmail.com>: > > > > > > Hi, > > > > > > TsFile API is not deprecated. In fact, it is designed for > this > > > scenario and > > > MapReduce/Spark computing. > > > > > > If you just use Reader and Writer API, there is something > you > > need to > > > know: > > > > > > Let's suppose your block size is x Bytes, > > (tsfile-format.properties: > > > group_size_in_byte). > > > > > > 1. If you write data and a shutdown occurs, then all data > that is > > > flushed > > > on disk is ok, and you can read the data ( class > > > org.apache.iotdb.tsfile.TsFileSequenceRead is an example, > but > > you need > > > to > > > change it a little. I think I can write an example.) > > > > > > 2. Actually, TsFile has the ability to allow you continue > to > > write > > > data at > > > the end of the incomplete file. However, We do not provide > this > > API > > > now... > > > If needed, I can add the API. > > > > > > 3. In this scenario, you will lose at most x Bytes data. > If you > > do not > > > accept that, something like WAL is needed. (It is not very > > complex, > > > but I > > > am not sure that whether it should be an embedded function > for > > TsFile). > > > > > > Up to now, we can consider that TsFile API is suitable for > your > > > scenario > > > (even though we need to add a little more API if you > desire). > > And you > > > can > > > get the ability to compress data, and query data from the > TsFile > > rather > > > than scan the data from the head to the tail. > > > > > > However, TsFile has one constraint: You can not write > > out-of-order data > > > into a TsFile, otherwise the query API may return > incomplete > > result. > > > But I think it is ok for real applications, because I do > not > > think > > > that a > > > device can generate out-of-order data.... > > > > > > For example, If you write two devices' data into one > TsFile, it > > is ok > > > if > > > you write data like: > > > - d1.t1, d1.t2, d2.t1, d2.t2, d2.t3, d1.t4, d1.t5 .... > > > or: > > > - d1.m1.t1, d1.m1.t2, d1.m2.t1, d1.m2.t2, d2.m1.t1 ... > > > > > > But you can not write data like: > > > - d1.m1.t2, d1.m1.t1 ... > > > > > > I think it is a good chance to improve TsFile to make it > more > > suitable > > > for > > > real applications, so please do not hesitate to tell me > more > > about > > > what you > > > think TsFile should want to have? > > > > > > Best, > > > ----------------------------------- > > > Xiangdong Huang > > > School of Software, Tsinghua University > > > > > > 黄向东 > > > 清华大学 软件学院 > > > > > > > > > Julian Feinauer <j.feina...@pragmaticminds.de> > 于2019年3月4日周一 > > 下午7:17写道： > > > > > > > Hi Xiangdong, > > > > > > > > thanks for the info. > > > > How is it in the case when you use the Reader / Writer > API for > > the > > > tsfiles > > > > directly (or should this be considered "deprecated")? > > > > Can these files come to corrupted state? > > > > > > > > One Situation where we have to deal with these > situations is > > "at the > > > edge" > > > > when we have devices inside large machines. > > > > Usually at the end of the shift these machines (and > therefore > > our > > > device) > > > > is powered off hard, so no shutdown or de-initialization > is > > possible. > > > > > > > > Best > > > > Julian > > > > > > > > Am 04.03.19, 12:14 schrieb "Xiangdong Huang" < > > saint...@gmail.com>: > > > > > > > > Hi, > > > > > > > > IoTDB can support either on a server with 7*24 or a > > RaspberryPi. > > > We > > > > have > > > > tested both the two scenario. > > > > > > > > When you shutdown an IoTDB instance in force (e.g., > power > > off) > > > and > > > > restart > > > > it again, no data loses ( if you enable the WAL). > > > > > > > > However, currently we do not optimize the time cost > of the > > > restart > > > > process. > > > > It is an important feature that we need to do, > because we > > hope > > > IoTDB > > > > can > > > > support data management either on the edge devices > or the > > data > > > center. > > > > > > > > And, the default configuration is not so suitable for > > running on > > > the > > > > edge > > > > device. (e.g., block size is 128MB, which is too > large for > > a > > > > RaspberryPi, > > > > and will slow down the restart process because there > are > > too > > > much WAL > > > > data > > > > on disk). > > > > > > > > Best, > > > > ----------------------------------- > > > > Xiangdong Huang > > > > School of Software, Tsinghua University > > > > > > > > 黄向东 > > > > 清华大学 软件学院 > > > > > > > > > > > > Tim Mitsch <t.mit...@pragmaticindustries.de> > 于2019年3月4日周一 > > > 下午6:53写道： > > > > > > > > > Hello development-team > > > > > > > > > > First of all thanks for developing this kind of > > interesting > > > project > > > > and > > > > > bringing it into apache incubator. > > > > > > > > > > I have a question regarding the place of operation > and > > > robustness: > > > > > > > > > > * Is iotDB concepted as application on a server > > which is > > > running > > > > 24/7 > > > > > or > > > > > * Is it also possible to run it on a device > like > > > RaspberryPi or > > > > IPC, > > > > > where operation can interrupt. > > > > > I’m asking because i’m searching for solution for a > > temporary > > > > storage that > > > > > is robust against spontaneous interrupt, e.g. > switch off > > > electricity > > > > > without regular shutdown of OS – have u tested > something > > like > > > this > > > > yet? > > > > > > > > > > Best regards > > > > > Tim > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
