Hi,
I have added a new TsFileIOWriter, which supports to recover data from a
broken TsFile (I mean, a TsFile that is not closed correctly). The PR is
https://github.com/apache/incubator-iotdb/pull/87.
You can use the new feature by:
```
NativeRestorableIOWriter rWriter = new NativeRestorableIOWriter(file);
TsFileWriter writer = new TsFileWriter(rWriter);
```
1. If the file is a complete TsFile (e.g., the FileMetadata has been
persistent into the file and the tail magic string is correct), then you
can not write data into this TsFile anymore (actually I can remove all
FileMetadata and then let you continue to write, but it needs more coding
work...), `TsFileWriter writer = new TsFileWriter(rWriter);` will throw a
IoException. You can check it before you create TsFileWriter by using
`rWriter.canWrite()`.
2. ChunkGroup is the basic unit that can be recover. That is, if a
ChunkGroup is complete, i.e., its chunkGroupFooter is complete, I will
remain it, otherwise I will truncate the chunk group. However, I can not
know whether a chunkGroupFooter is complete (if only one byte lose, the
deserialize method does not throw exception, because something like
`deserializeToInt(inputstream)` does not throw exception (it will use 3
bytes to convert them to a integer). So, when I say "if a ChunkGroup is
complete", I mean that not only its chunkGroupFooter is serialized
successfully, but also there is at least one more byte is serialized into
the file followed the footer.
3. I have tested many cases, such only a magic head string is persisted,
only a ChunkHeader is persisted, the first ChunkHeader is not persisted
completely, only some Chunks are persisted, a ChunkGroupFooter is
persisted, Two ChunkGroupFooter are persisted, and a complete TsFile is
persisted, etc.. (`NativeRestorableIOWriterTest.java`, hope I have cover
all cases)
Hope TsFile can be used in edge device management applications ASAP!
Now please enjoy the new feature.
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Julian Feinauer <[email protected]> 于2019年3月5日周二 下午8:10写道：
> Hey,
>
> thank you fort he link... I did not know of this.. this is exactly what I
> was looking for!
>
> Julian
>
> PS.: Looking forward to your PR : )
>
> Am 05.03.19, 12:26 schrieb "Xiangdong Huang" <[email protected]>:
>
> Hi,
>
> 1. We have a document to introduce that:
> https://cwiki.apache.org/confluence/display/IOTDB/TsFile+Format
>
> 2. The new API for recovering data is almost done. I am writing the UTs
> now. Maybe I can submit a PR tonight (if everything is fine...)
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Julian Feinauer <[email protected]> 于2019年3月5日周二 下午6:00写道：
>
> > Hi Xiangdong,
> >
> > that sounds excellent.
> > Do you have a short overview of how the file format is designed on
> disk?
> > I know that its somewhat similar to parquet but I did not find more
> > details.
> > Basically what would suffice for us would be something like skipping
> an
> > invalid column group (or how you name it) and go on with the next,
> or so.
> >
> > Julian
> >
> > Am 04.03.19, 13:21 schrieb "Xiangdong Huang" <[email protected]>:
> >
> > Hi,
> >
> > If so, I think I need to add a new API to allow you continue to
> write
> > data
> > in an existing but not closed correctly TsFile. Then everything
> is
> > fine
> > for you :D
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Julian Feinauer <[email protected]> 于2019年3月4日周一
> 下午8:08写道：
> >
> > > Hey Xiangdong,
> > >
> > > thanks for the great explanation.
> > > And in fact, I agree with you that it would be best if we
> start to
> > play
> > > around with it and reply all our findings or wishes back to
> this
> > list (in
> > > fact that proved to be beneficial in plc4x as well).
> > >
> > > You confirm my thoughts about the two "levels" of APIs (DB and
> file)
> > and
> > > the file api is exactly what we looked for for our use case.
> > > As we do not care much about data loss (when an edge device
> fails
> > its...
> > > gone).
> > > The crucial point for us is that no corrupt files can be
> generated.
> > > This means I'm fine when the last data submitted is lost but
> I'm not
> > fine
> > > if we can get to a situation where the last datafile is
> completely
> > lost
> > > (well, perhaps this could be acceptable).
> > >
> > > @tim: Perhaps its best when you give some more information to
> > Xiangdong
> > > about our idea, and we can also point to our current code in
> github
> > >
> > > Julian
> > >
> > > Am 04.03.19, 13:03 schrieb "Xiangdong Huang" <
> [email protected]>:
> > >
> > > Hi,
> > >
> > > TsFile API is not deprecated. In fact, it is designed for
> this
> > > scenario and
> > > MapReduce/Spark computing.
> > >
> > > If you just use Reader and Writer API, there is something
> you
> > need to
> > > know:
> > >
> > > Let's suppose your block size is x Bytes,
> > (tsfile-format.properties:
> > > group_size_in_byte).
> > >
> > > 1. If you write data and a shutdown occurs, then all data
> that is
> > > flushed
> > > on disk is ok, and you can read the data ( class
> > > org.apache.iotdb.tsfile.TsFileSequenceRead is an example,
> but
> > you need
> > > to
> > > change it a little. I think I can write an example.)
> > >
> > > 2. Actually, TsFile has the ability to allow you continue
> to
> > write
> > > data at
> > > the end of the incomplete file. However, We do not provide
> this
> > API
> > > now...
> > > If needed, I can add the API.
> > >
> > > 3. In this scenario, you will lose at most x Bytes data.
> If you
> > do not
> > > accept that, something like WAL is needed. (It is not very
> > complex,
> > > but I
> > > am not sure that whether it should be an embedded function
> for
> > TsFile).
> > >
> > > Up to now, we can consider that TsFile API is suitable for
> your
> > > scenario
> > > (even though we need to add a little more API if you
> desire).
> > And you
> > > can
> > > get the ability to compress data, and query data from the
> TsFile
> > rather
> > > than scan the data from the head to the tail.
> > >
> > > However, TsFile has one constraint: You can not write
> > out-of-order data
> > > into a TsFile, otherwise the query API may return
> incomplete
> > result.
> > > But I think it is ok for real applications, because I do
> not
> > think
> > > that a
> > > device can generate out-of-order data....
> > >
> > > For example, If you write two devices' data into one
> TsFile, it
> > is ok
> > > if
> > > you write data like:
> > > - d1.t1, d1.t2, d2.t1, d2.t2, d2.t3, d1.t4, d1.t5 ....
> > > or:
> > > - d1.m1.t1, d1.m1.t2, d1.m2.t1, d1.m2.t2, d2.m1.t1 ...
> > >
> > > But you can not write data like:
> > > - d1.m1.t2, d1.m1.t1 ...
> > >
> > > I think it is a good chance to improve TsFile to make it
> more
> > suitable
> > > for
> > > real applications, so please do not hesitate to tell me
> more
> > about
> > > what you
> > > think TsFile should want to have?
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > > 黄向东
> > > 清华大学 软件学院
> > >
> > >
> > > Julian Feinauer <[email protected]>
> 于2019年3月4日周一
> > 下午7:17写道：
> > >
> > > > Hi Xiangdong,
> > > >
> > > > thanks for the info.
> > > > How is it in the case when you use the Reader / Writer
> API for
> > the
> > > tsfiles
> > > > directly (or should this be considered "deprecated")?
> > > > Can these files come to corrupted state?
> > > >
> > > > One Situation where we have to deal with these
> situations is
> > "at the
> > > edge"
> > > > when we have devices inside large machines.
> > > > Usually at the end of the shift these machines (and
> therefore
> > our
> > > device)
> > > > is powered off hard, so no shutdown or de-initialization
> is
> > possible.
> > > >
> > > > Best
> > > > Julian
> > > >
> > > > Am 04.03.19, 12:14 schrieb "Xiangdong Huang" <
> > [email protected]>:
> > > >
> > > > Hi,
> > > >
> > > > IoTDB can support either on a server with 7*24 or a
> > RaspberryPi.
> > > We
> > > > have
> > > > tested both the two scenario.
> > > >
> > > > When you shutdown an IoTDB instance in force (e.g.,
> power
> > off)
> > > and
> > > > restart
> > > > it again, no data loses ( if you enable the WAL).
> > > >
> > > > However, currently we do not optimize the time cost
> of the
> > > restart
> > > > process.
> > > > It is an important feature that we need to do,
> because we
> > hope
> > > IoTDB
> > > > can
> > > > support data management either on the edge devices
> or the
> > data
> > > center.
> > > >
> > > > And, the default configuration is not so suitable for
> > running on
> > > the
> > > > edge
> > > > device. (e.g., block size is 128MB, which is too
> large for
> > a
> > > > RaspberryPi,
> > > > and will slow down the restart process because there
> are
> > too
> > > much WAL
> > > > data
> > > > on disk).
> > > >
> > > > Best,
> > > > -----------------------------------
> > > > Xiangdong Huang
> > > > School of Software, Tsinghua University
> > > >
> > > > 黄向东
> > > > 清华大学 软件学院
> > > >
> > > >
> > > > Tim Mitsch <[email protected]>
> 于2019年3月4日周一
> > > 下午6:53写道：
> > > >
> > > > > Hello development-team
> > > > >
> > > > > First of all thanks for developing this kind of
> > interesting
> > > project
> > > > and
> > > > > bringing it into apache incubator.
> > > > >
> > > > > I have a question regarding the place of operation
> and
> > > robustness:
> > > > >
> > > > > * Is iotDB concepted as application on a server
> > which is
> > > running
> > > > 24/7
> > > > > or
> > > > > * Is it also possible to run it on a device
> like
> > > RaspberryPi or
> > > > IPC,
> > > > > where operation can interrupt.
> > > > > I’m asking because i’m searching for solution for a
> > temporary
> > > > storage that
> > > > > is robust against spontaneous interrupt, e.g.
> switch off
> > > electricity
> > > > > without regular shutdown of OS – have u tested
> something
> > like
> > > this
> > > > yet?
> > > > >
> > > > > Best regards
> > > > > Tim
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
