[
https://issues.apache.org/jira/browse/HDFS-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285567#comment-13285567
]
Ivan Kelly commented on HDFS-3470:
----------------------------------
This occurs because of the way we open ledgers in bookkeeper. The bookies keep
a record of the last entry which has been confirmed at the client (the client
piggybacks this on newer entries). A client coming to open a ledger which
hasn't been closed will get the last confirmed entry, and then read forward,
one entry at a time, to find the last entry.
Perhaps we should change the error to a warn, no move where it bad read is
logged. In a normal read, it's bad, but for recovery, it's expected. Moving to
bookkeeper in any case, as this can't be changed on the HDFS side.
> [Log Improvement]: Error logs logged at the time of switch which is misleading
> ------------------------------------------------------------------------------
>
> Key: HDFS-3470
> URL: https://issues.apache.org/jira/browse/HDFS-3470
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: suja s
> Priority: Minor
>
> For every switch the following error is logged at NN side
> 2012-05-17 11:19:46,021 ERROR org.apache.bookkeeper.client.PendingReadOp:
> Error: No such entry while reading entry: 1 ledgerId: 16 from bookie:
> /XX.XX.XX.55:3182
> 2012-05-17 11:19:46,022 ERROR org.apache.bookkeeper.client.PendingReadOp:
> Error: No such entry while reading entry: 1 ledgerId: 16 from bookie:
> /XX.XX.XX.55:3183
> 2012-05-17 11:19:46,022 ERROR org.apache.bookkeeper.client.PendingReadOp:
> Error: No such entry while reading entry: 1 ledgerId: 16 from bookie:
> /XX.XX.XX.55:3181
> {noformat}
> 2012-05-17 11:19:45,891 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services
> started for standby state
> 2012-05-17 11:19:45,893 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Edit log tailer
> interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:329)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:274)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:291)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:512)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:287)
> 2012-05-17 11:19:45,895 INFO
> org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie:
> /XX.XX.XX.55:3182
> 2012-05-17 11:19:45,895 INFO
> org.apache.bookkeeper.proto.PerChannelBookieClient: Disconnected from bookie:
> /XX.XX.XX.55:3181
> 2012-05-17 11:19:45,913 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x1375944cf250027 closed
> 2012-05-17 11:19:45,913 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2012-05-17 11:19:45,913 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
> 2012-05-17 11:19:45,913 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=XX.XX.XX.55:2182 sessionTimeout=3000
> watcher=org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager$ZkConnectionWatcher@734d246
> 2012-05-17 11:19:45,914 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server /XX.XX.XX.55:2182
> 2012-05-17 11:19:45,915 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to HOST-10-18-52-55/XX.XX.XX.55:2182, initiating
> session
> 2012-05-17 11:19:45,947 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server HOST-10-18-52-55/XX.XX.XX.55:2182, sessionid
> = 0x1375944cf250029, negotiated timeout = 4000
> 2012-05-17 11:19:45,994 INFO
> org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to
> bookie: /XX.XX.XX.55:3181
> 2012-05-17 11:19:45,996 INFO
> org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to
> bookie: /XX.XX.XX.55:3183
> 2012-05-17 11:19:46,001 INFO
> org.apache.bookkeeper.proto.PerChannelBookieClient: Successfully connected to
> bookie: /XX.XX.XX.55:3182
> 2012-05-17 11:19:46,021 ERROR org.apache.bookkeeper.client.PendingReadOp:
> Error: No such entry while reading entry: 1 ledgerId: 16 from bookie:
> /XX.XX.XX.55:3182
> 2012-05-17 11:19:46,022 ERROR org.apache.bookkeeper.client.PendingReadOp:
> Error: No such entry while reading entry: 1 ledgerId: 16 from bookie:
> /XX.XX.XX.55:3183
> 2012-05-17 11:19:46,022 ERROR org.apache.bookkeeper.client.PendingReadOp:
> Error: No such entry while reading entry: 1 ledgerId: 16 from bookie:
> /XX.XX.XX.55:3181
> 2012-05-17 11:19:46,045 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/May8/hadoop-3.0.0-SNAPSHOT/hadoop-root/dfs/name/current
> 2012-05-17 11:19:46,049 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /home/May8/hadoop-3.0.0-SNAPSHOT/hadoop-root/dfs/name/current/edits_inprogress_0000000000000000036
> ->
> /home/May8/hadoop-3.0.0-SNAPSHOT/hadoop-root/dfs/name/current/edits_0000000000000000036-0000000000000000036
> 2012-05-17 11:19:46,049 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Catching up to latest
> edits from old active before taking over writer role in edits logs.
> 2012-05-17 11:19:46,077 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
> Reading
> org.apache.hadoop.contrib.bkjournal.BookKeeperEditLogInputStream@38f0b51d
> expecting start txid #37
> 2012-05-17 11:19:46,128 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
> Edits file
> BookKeeper[org.apache.bookkeeper.client.ReadOnlyLedgerHandle@615e7597,first=37,last=37]
> of size 13 edits # 1 loaded in 0 seconds.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
