[
https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-8497:
---------------------------------------
Attachment: HBASE-8497-v0.patch
Attached is a patch for adding trailer at the end of a WAL file.
Currently, it is a empty trailer. Additional fields can be added to the
WALTrailer message in WAL.proto. I tested it by running a patched cluster on my
local system; ran TestHLog* tests.
Some considerations while working on the patch:
1) Trailer is optional. WAL should be readable if it's not present. If it is
present, then WAL is read to the point just before trailer starts.
2) WAL is structured as:
<WALHeader><WALEdits>...<WALEdits><Trailer><Trailer_Size><WAL_COMPLETE_MAGIC>
3) Trailer is set only at the close call. There are APIs in HLog to set/get
trailer.
4) Files under recovery/incorrect length/reading files when they are not closed:
There may be case when a reader is opened even when the log file is not closed:
ReplicationLogReader, for example. Or,
If there is trailer but we couldn't get correct Wal size due to some HDFS issue
(under recovery, etc) while opening.
In such cases, the trailer is not read by the reader. While reading the wal, if
ProtobufLogReader encounters the trailer in the next() call, it returns false,
which is a legit behavior. I added unit tests to simulate this behavior.
> Protobuf WAL also needs a trailer
> ----------------------------------
>
> Key: HBASE-8497
> URL: https://issues.apache.org/jira/browse/HBASE-8497
> Project: HBase
> Issue Type: Sub-task
> Components: Protobufs, wal
> Affects Versions: 0.95.1
> Reporter: Enis Soztutar
> Fix For: 0.98.0, 0.95.1
>
> Attachments: HBASE-8497-v0.patch
>
>
> New Protobuf WAL has a header, but we will probably need a trailer as well,
> reserved for later usage.
> Right now, we can we just serialize an empty trailer, but putting more
> metadata there, like range of sequence_id's, region names, table names etc
> might be needed in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
