[jira] [Updated] (HDFS-13236) Standby NN down with error encountered while tailing edits

2018-07-04 Thread Yuriy Malygin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuriy Malygin updated HDFS-13236:
-
Description: 
After update Hadoop from 2.7.3 to 3.0.0 standby NN down with error encountered 
while tailing edits from JN:
{code:java}
Feb 28 01:58:31 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:31,594 INFO 
[FSImageSaver for /one/hadoop-data/dfs of type IMAGE_AND_EDITS] 
FSImageFormatProtobuf - Image file 
/one/hadoop-data/dfs/current/fsimage.ckpt_012748979
98 of size 4595971949 bytes saved in 93 seconds.
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Going to retain 2 
images with txid >= 1274897935
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Purging old image 
FSImageFile(file=/one/hadoop-data/dfs/current/fsimage_01274897875, 
cpktTxId
=01274897875)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Reading 
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@6a168e6f 
expecting start txid #1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Start loading edits file 
http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A10
56233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true, 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848;
inProgressOk=true
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999
torageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true,
 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217
-aef5-6ed206893848=true' to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true'
 to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,680 ERROR 
[Edit log tailer] FSEditLogLoader - Encountered exception on operation AddOp 
[length=0, inodeId=145550319, 
path=/kafka/parquet/infrastructureGrace/date=2018-02-28/_temporary/1/_temporary/attempt_1516181147167_20856_r_98_0/part-r-00098.gz.parquet,
 replication=3, mtime=1519772206615, atime=1519772206615, blockSize=134217728, 
blocks=[], permissions=root:supergroup:rw-r--r--, aclEntries=null, 
clientName=DFSClient_attempt_1516181147167_20856_r_98_0_1523538799_1, 
clientMachine=10.137.2.142, overwrite=false, RpcClientId=, RpcCallId=271996603, 
storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_ADD, txid=1274898002]
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 
java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected 
length 16
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:946)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 

[jira] [Updated] (HDFS-13236) Standby NN down with error encountered while tailing edits

2018-03-05 Thread Yuriy Malygin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuriy Malygin updated HDFS-13236:
-
Description: 
After update Hadoop from 2.7.3 to 3.0.0 standby NN down with error encountered 
while tailing edits from JN:
{code:java}
Feb 28 01:58:31 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:31,594 INFO 
[FSImageSaver for /one/hadoop-data/dfs of type IMAGE_AND_EDITS] 
FSImageFormatProtobuf - Image file 
/one/hadoop-data/dfs/current/fsimage.ckpt_012748979
98 of size 4595971949 bytes saved in 93 seconds.
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Going to retain 2 
images with txid >= 1274897935
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Purging old image 
FSImageFile(file=/one/hadoop-data/dfs/current/fsimage_01274897875, 
cpktTxId
=01274897875)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Reading 
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@6a168e6f 
expecting start txid #1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Start loading edits file 
http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A10
56233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true, 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848;
inProgressOk=true
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999
torageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true,
 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217
-aef5-6ed206893848=true' to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true'
 to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,680 ERROR 
[Edit log tailer] FSEditLogLoader - Encountered exception on operation AddOp 
[length=0, inodeId=145550319, 
path=/kafka/parquet/infrastructureGrace/date=2018-02-28/_temporary/1/_temporary/attempt_1516181147167_20856_r_98_0/part-r-00098.gz.parquet,
 replication=3, mtime=1519772206615, atime=1519772206615, blockSize=134217728, 
blocks=[], permissions=root:supergroup:rw-r--r--, aclEntries=null, 
clientName=DFSClient_attempt_1516181147167_20856_r_98_0_1523538799_1, 
clientMachine=10.137.2.142, overwrite=false, RpcClientId=, RpcCallId=271996603, 
storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_ADD, txid=1274898002]
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 
java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected 
length 16
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:946)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 

[jira] [Updated] (HDFS-13236) Standby NN down with error encountered while tailing edits

2018-03-05 Thread Yuriy Malygin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuriy Malygin updated HDFS-13236:
-
Description: 
After update Hadoop from 2.7.3 to 3.0.0 standby NN down with error encountered 
while tailing edits from JN:
{code:java}
Feb 28 01:58:31 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:31,594 INFO 
[FSImageSaver for /one/hadoop-data/dfs of type IMAGE_AND_EDITS] 
FSImageFormatProtobuf - Image file 
/one/hadoop-data/dfs/current/fsimage.ckpt_012748979
98 of size 4595971949 bytes saved in 93 seconds.
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Going to retain 2 
images with txid >= 1274897935
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Purging old image 
FSImageFile(file=/one/hadoop-data/dfs/current/fsimage_01274897875, 
cpktTxId
=01274897875)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Reading 
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@6a168e6f 
expecting start txid #1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Start loading edits file 
http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A10
56233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true, 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848;
inProgressOk=true
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999
torageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true,
 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217
-aef5-6ed206893848=true' to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true'
 to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,680 ERROR 
[Edit log tailer] FSEditLogLoader - Encountered exception on operation AddOp 
[length=0, inodeId=145550319, 
path=/kafka/parquet/infrastructureGrace/date=2018-02-28/_temporary/1/_temporary/attempt_1516181147167_20856_r_98_0/part-r-00098.gz.parquet,
 replication=3, mtime=1519772206615, atime=1519772206615, blockSize=134217728, 
blocks=[], permissions=root:supergroup:rw-r--r--, aclEntries=null, 
clientName=DFSClient_attempt_1516181147167_20856_r_98_0_1523538799_1, 
clientMachine=10.137.2.142, overwrite=false, RpcClientId=, RpcCallId=271996603, 
storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_ADD, txid=1274898002]
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 
java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected 
length 16
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:946)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 

[jira] [Updated] (HDFS-13236) Standby NN down with error encountered while tailing edits

2018-03-05 Thread Yuriy Malygin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuriy Malygin updated HDFS-13236:
-
Description: 
After update Hadoop from 2.7.3 to 3.0.0 standby NN down with error encountered 
while tailing edits from JN:
{code:java}
Feb 28 01:58:31 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:31,594 INFO 
[FSImageSaver for /one/hadoop-data/dfs of type IMAGE_AND_EDITS] 
FSImageFormatProtobuf - Image file 
/one/hadoop-data/dfs/current/fsimage.ckpt_012748979
98 of size 4595971949 bytes saved in 93 seconds.
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Going to retain 2 
images with txid >= 1274897935
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Purging old image 
FSImageFile(file=/one/hadoop-data/dfs/current/fsimage_01274897875, 
cpktTxId
=01274897875)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Reading 
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@6a168e6f 
expecting start txid #1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Start loading edits file 
http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A10
56233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true, 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848;
inProgressOk=true
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999
torageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true,
 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217
-aef5-6ed206893848=true' to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true'
 to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,680 ERROR 
[Edit log tailer] FSEditLogLoader - Encountered exception on operation AddOp 
[length=0, inodeId=145550319, 
path=/kafka/parquet/infrastructureGrace/date=2018-02-28/_temporary/1/_temporary/attempt_1516181147167_20856_r_98_0/part-r-00098.gz.parquet,
 replication=3, mtime=1519772206615, atime=1519772206615, blockSize=134217728, 
blocks=[], permissions=root:supergroup:rw-r--r--, aclEntries=null, 
clientName=DFSClient_attempt_1516181147167_20856_r_98_0_1523538799_1, 
clientMachine=10.137.2.142, overwrite=false, RpcClientId=, RpcCallId=271996603, 
storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_ADD, txid=1274898002]
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 
java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected 
length 16
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:946)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at