[
https://issues.apache.org/jira/browse/HDFS-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509948#comment-13509948
]
Colin Patrick McCabe commented on HDFS-4045:
--------------------------------------------
Here's a proposal that might make this a little nicer:
* augment the {{StreamLimiter}} interface with a {{getRemainingBeforeLimit}}
call. This call will return how many bytes are left before the limit hits.
* add a field, {{decodedSize}}, to {{FSEDitLogOp.}} At the end of
{{FSEditLogOp#decodeOp}}, set {{this.decodedSize =
limiter.getRemainingBeforeLimit()}}.
* add a {{FSEditLogOp#decodeRawOp}} function which returns a byte array. It
does so by calling readOp in a loop, and noticing the decoded opcode sizes.
Then it just pulls the bytes from the stream itself.
I think this is nicer because it doesn't burden the main opcode loading path
with the need to re-read every byte twice or create an array with all the
opcode bytes each time. Calculating the remaining bytes before the limit is
really cheap, just a subtraction. {{decodeRawOp}} still does re-read
everything twice, but this only affects that one code path, not all of them.
For extra efficiency, decodeRawOp could just fill an array you pass, rather
than creating a lot of garbage all the time. Allocating a single 1.5 MB array
is not a big deal.
Thoughts?
> SecondaryNameNode cannot read from QuorumJournal URI
> ----------------------------------------------------
>
> Key: HDFS-4045
> URL: https://issues.apache.org/jira/browse/HDFS-4045
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.0.0
> Reporter: Vinithra Varadharajan
> Assignee: Andy Isaacson
> Attachments: hdfs-4045-2.txt, hdfs4045-3.txt, hdfs4045-4.txt,
> hdfs-4045.txt
>
>
> If HDFS is set up in basic mode (non-HA) with QuorumJournal, and the
> dfs.namenode.edits.dir is set to only the QuorumJournal URI and no local dir,
> the SecondaryNameNode is unable to do a checkpoint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
