[
https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385131#comment-15385131
]
Jiayi Zhou commented on HDFS-10519:
-----------------------------------
Thanks Andrew for the review. I have attached a new patch to solve the problems.
* Config key name fixed.
* Comment added in hdfs-default.xml. Users want to turn it on if they want more
up-to-date data on SBNN.
bq. EditLogTailer, regarding the comment: edits themselves are not in-progress,
an edit log segment is. Consider renaming the boolean also.
* Fixed. Rename to inProgressOk which is just the parameter name in
selectInputStream()
bq. "isTail" is also not very descriptive, what it's really doing is bounding
the tailing to the committed txID length right?
* Change it to isBoundedByDurableTxId. It looks a bit long. Any suggestions?
Journal.java
* Unused boolean has been removed
* Rename to getCommittedTxnId
* Comment added. numTxn = 0 implies it's a dummy edits send, so we can return
early.
bq. JournalSet, why is the correct committed txnid to set here 0 rather than
the max txId from the set of logs?
* Good point. I lazily set it to 0 since it would not be used. Change it to the
max txId from the logs.
QuorumOutputStream
* Fixed. Boolean has been changed to updateCommittedTxnId
* Comments added. Committed txnId could be one-batch stale if we don't have
this dummy flush. A thread could be added to do this dummy flush, but setting a
proper time period needs to be considered. For now, I prefer to leave it sync.
bq. QuorumJournalManager, the ternary can be written instead with Math.min yea?
I'd prefer to see a little logic rework that avoids the need for the else also.
* Fixed.
RemoteEditLogManifest
* Remove single-arg form of constructor.
* Add a validation in checkState. committedTxnId should be larger than the
start TxId of the logs.
* toString() modified.
bq. TestBKSR, TestFJM, you don't need to modify the call since we kept the old
overload.
* Fixed. In TestFJM, parameter is JournalManager, changed it to
FileJournalManager.
bq. TestNNSRManager, we should pass through the parameter in the mock, like the
other args
* Fixed.
TestStandbyInProgressTail
* Typos fixed
* Thread.sleep removed. Use manual doTailEdits() instead.
* @Before and @After methods added. Null check is included in tearDown()
* Assert methods are static now.
* Randomized tests could be tricky. I haven't come up with a good idea yet. A
new jira might be filed for it in the future.
> Add a configuration option to enable in-progress edit log tailing
> -----------------------------------------------------------------
>
> Key: HDFS-10519
> URL: https://issues.apache.org/jira/browse/HDFS-10519
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha
> Reporter: Jiayi Zhou
> Assignee: Jiayi Zhou
> Priority: Minor
> Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch,
> HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch,
> HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch
>
>
> Standby Namenode has the option to do in-progress edit log tailing to improve
> the data freshness. In-progress tailing is already implemented, but it's not
> enabled as default configuration. And there's no related configuration key to
> turn it on.
> Adding a related configuration key to let Standby Namenode is reasonable and
> would be a basis for further improvement on Standby Namenode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
