There have been multiple reasons of the region server jvm crash. For one of
such errors, the logs are attached. Let me know your inputs.
Thanks,
Amit.
On Thu, Mar 31, 2016 at 6:15 PM, Mujtaba Chohan <mujt...@apache.org> wrote:
> Can you attached last couple of hundred lines from RS log before it
> crashed? Also what's the RS heap size?
>
>
> On Thu, Mar 31, 2016 at 1:48 AM, Amit Shah <amits...@gmail.com> wrote:
>
>> Hi,
>>
>> We have been experimenting hbase (version 1.0) and phoenix (version 4.6)
>> for our OLAP workload. In order to precalculate aggregates we have been
>> executing an upsert phoenix query that aggregates raw data (over 10 mil
>> records) to generate an OLAP cube.
>>
>> While executing the query, one of the region servers in a cluster of 3 RS
>> crashes. I am trying to figure out what could be causing the region server
>> to crash.
>> The server shows high disk operations before the jvm crashed. Kindly find
>> the disk and other stats attached.
>>
>> Any suggestions on where could I look into would be helpful.
>>
>> The upsert query that was executed is
>>
>> upsert into AGENT_TER_PRO
>> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT)
>> select /*+ INDEX(TRANSACTIONS AG_TER_PRO2) */
>> AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES)
>> SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD)
>> SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT) AVG_PRICE_PER_UNIT from
>> TRANSACTIONS group by AGENT_ID,TERRITORY_ID,PRODUCT_ID;
>>
>> Thanks,
>> Amit.
>>
>>
>
2016-03-31 09:31:59,570 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 127.0.0.1,
dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS, blockid: 1073783872, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, success: true
2016-03-31 09:32:28,839 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788390_47741 src:
/10.210.26.97:56968 dest: /10.210.26.106:50010
2016-03-31 09:32:28,847 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:56968, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_680257937_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788390_47741, duration: 7013280
2016-03-31 09:32:28,847 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788390_47741,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2016-03-31 09:32:35,895 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788390_47741 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788390
for deletion
2016-03-31 09:32:35,895 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788390_47741 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788390
2016-03-31 09:33:57,505 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788391_47742 src:
/10.210.26.97:56977 dest: /10.210.26.106:50010
2016-03-31 09:33:57,913 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:56977, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_255100059_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788391_47742, duration: 7442331
2016-03-31 09:33:57,913 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788391_47742,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2016-03-31 09:34:05,895 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788391_47742 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788391
for deletion
2016-03-31 09:34:05,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788391_47742 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788391
2016-03-31 09:35:27,607 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788392_47743 src:
/10.210.26.20:62135 dest: /10.210.26.106:50010
2016-03-31 09:35:27,619 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.20:62135, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-57727167_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788392_47743, duration:
10020731
2016-03-31 09:35:27,619 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788392_47743,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2016-03-31 09:35:29,895 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788392_47743 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788392
for deletion
2016-03-31 09:35:29,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788392_47743 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788392
2016-03-31 09:36:25,057 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 127.0.0.1,
dest: 127.0.0.1, op: RELEASE_SHORT_CIRCUIT_FDS, shmId:
998e10f054dcba85ccc331ade0f31293, slotIdx: 23, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, success: true
2016-03-31 09:36:28,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788393_47744 src:
/10.210.26.97:56990 dest: /10.210.26.106:50010
2016-03-31 09:36:28,337 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:56990, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-1128436266_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788393_47744, duration:
11266163
2016-03-31 09:36:28,337 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788393_47744,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2016-03-31 09:36:32,897 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788393_47744 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788393
for deletion
2016-03-31 09:36:32,898 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788393_47744 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788393
2016-03-31 09:37:30,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788394_47745 src:
/10.210.26.20:62147 dest: /10.210.26.106:50010
2016-03-31 09:37:30,515 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.20:62147, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-2099132308_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788394_47745, duration:
50966124
2016-03-31 09:37:30,515 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788394_47745,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2016-03-31 09:37:32,895 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788394_47745 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788394
for deletion
2016-03-31 09:37:32,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788394_47745 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788394
2016-03-31 09:38:03,961 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for BP-1253436086-10.210.25.47-1456383662287:blk_1073788354_47705
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:471)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:857)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:797)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
at java.lang.Thread.run(Thread.java:745)
2016-03-31 09:38:03,962 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788354_47705,
type=HAS_DOWNSTREAM_IN_PIPELINE: Thread is interrupted.
2016-03-31 09:38:03,962 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788354_47705,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:38:03,962 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
opWriteBlock BP-1253436086-10.210.25.47-1456383662287:blk_1073788354_47705
received exception java.io.IOException: Premature EOF from inputStream
2016-03-31 09:38:03,963 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
v-in-sd-hack-04.synygy.net:50010:DataXceiver error processing WRITE_BLOCK
operation src: /10.210.26.106:56629 dst: /10.210.26.106:50010
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:471)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:857)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:797)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
at java.lang.Thread.run(Thread.java:745)
2016-03-31 09:38:17,896 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
NameNode at v-in-sd-hack-03.synygy.net/10.210.25.47:8022 calls
recoverBlock(BP-1253436086-10.210.25.47-1456383662287:blk_1073788354_47705,
targets=[DatanodeInfoWithStorage[10.210.26.106:50010,null,null],
DatanodeInfoWithStorage[10.210.26.97:50010,null,null],
DatanodeInfoWithStorage[10.210.26.20:50010,null,null]],
newGenerationStamp=47746)
2016-03-31 09:38:17,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
initReplicaRecovery: blk_1073788354_47705, recoveryId=47746,
replica=ReplicaBeingWritten, blk_1073788354_47705, RBW
getNumBytes() = 1231
getBytesOnDisk() = 1231
getVisibleLength()= 1231
getVolume() = /dfs/dn/current
getBlockFile() =
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/rbw/blk_1073788354
bytesAcked=1231
bytesOnDisk=1231
2016-03-31 09:38:17,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
initReplicaRecovery: changing replica state for blk_1073788354_47705 from RBW
to RUR
2016-03-31 09:38:17,903 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
updateReplica: BP-1253436086-10.210.25.47-1456383662287:blk_1073788354_47705,
recoveryId=47746, length=1231, replica=ReplicaUnderRecovery,
blk_1073788354_47705, RUR
getNumBytes() = 1231
getBytesOnDisk() = 1231
getVisibleLength()= 1231
getVolume() = /dfs/dn/current
getBlockFile() =
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/rbw/blk_1073788354
recoveryId=47746
original=ReplicaBeingWritten, blk_1073788354_47705, RBW
getNumBytes() = 1231
getBytesOnDisk() = 1231
getVisibleLength()= 1231
getVolume() = /dfs/dn/current
getBlockFile() =
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/rbw/blk_1073788354
bytesAcked=1231
bytesOnDisk=1231
2016-03-31 09:38:20,066 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788395_47747 src:
/10.210.26.97:57005 dest: /10.210.26.106:50010
2016-03-31 09:38:20,067 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788397_47749 src:
/10.210.26.97:57006 dest: /10.210.26.106:50010
2016-03-31 09:38:20,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788396_47748 src:
/10.210.26.97:57007 dest: /10.210.26.106:50010
2016-03-31 09:38:20,072 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:57005, dest: /10.210.26.106:50010, bytes: 389, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-786337495_1, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788395_47747, duration: 2882950
2016-03-31 09:38:20,073 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788395_47747,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:38:20,074 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:57006, dest: /10.210.26.106:50010, bytes: 386, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-786337495_1, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788397_47749, duration: 4188738
2016-03-31 09:38:20,074 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788397_47749,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:38:20,076 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:57007, dest: /10.210.26.106:50010, bytes: 384, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-786337495_1, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788396_47748, duration: 4515330
2016-03-31 09:38:20,076 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788396_47748,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:38:20,093 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788398_47750 src:
/10.210.26.97:57009 dest: /10.210.26.106:50010
2016-03-31 09:38:20,098 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:57009, dest: /10.210.26.106:50010, bytes: 374, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-786337495_1, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788398_47750, duration: 2493829
2016-03-31 09:38:20,099 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788398_47750,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:38:27,503 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788399_47751 src:
/10.210.26.97:57011 dest: /10.210.26.106:50010
2016-03-31 09:38:27,518 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.26.97:57011, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-2022766440_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788399_47751, duration:
11237360
2016-03-31 09:38:27,518 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788399_47751,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:38:32,897 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788399_47751 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788399
for deletion
2016-03-31 09:38:32,897 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788399_47751 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788399
2016-03-31 09:39:32,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788354_47746 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788354
for deletion
2016-03-31 09:39:32,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788354_47746 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788354
2016-03-31 09:39:57,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788400_47752 src:
/10.210.25.47:60660 dest: /10.210.26.106:50010
2016-03-31 09:39:57,487 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.25.47:60660, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_16276458_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788400_47752, duration: 9495623
2016-03-31 09:39:57,488 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788400_47752,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:40:05,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788400_47752 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788400
for deletion
2016-03-31 09:40:05,897 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788400_47752 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788400
2016-03-31 09:41:29,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1253436086-10.210.25.47-1456383662287:blk_1073788401_47753 src:
/10.210.25.47:60740 dest: /10.210.26.106:50010
2016-03-31 09:41:29,030 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.210.25.47:60740, dest: /10.210.26.106:50010, bytes: 56, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-72983164_64, offset: 0, srvID:
96de5998-f972-4568-a7d5-3391d9a37c71, blockid:
BP-1253436086-10.210.25.47-1456383662287:blk_1073788401_47753, duration:
10149657
2016-03-31 09:41:29,030 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1253436086-10.210.25.47-1456383662287:blk_1073788401_47753,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2016-03-31 09:41:32,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073788401_47753 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788401
for deletion
2016-03-31 09:41:32,896 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1253436086-10.210.25.47-1456383662287 blk_1073788401_47753 file
/dfs/dn/current/BP-1253436086-10.210.25.47-1456383662287/current/finalized/subdir0/subdir181/blk_1073788401
