Re:

2018-06-13 Thread Sajid Mohammed 
what is your rack topology ?

On Tue, Jun 12, 2018 at 9:26 PM Sowjanya Kakarala 
wrote:

> Hi Guys,
>
>
> I have 4datanodes and one master node EMR cluster with 120GB data storage
> left. I have been running sqoop jobs which loads data to hive table. After
> some jobs ran successfully I suddenly see these errors all over the name
> node logs and datanodes logs.
>
> I have tried changing so many configurations as suggeted in stackoverflow
> and hortonworks sites but couldnt find a way for fixing it.
>
>
> Here is the error:
>
> 2018-06-12 15:32:35,933 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /user/hive/warehouse/monolith.db/tblname/_SCRATCH0.28417629602676764/time_stamp=2018-04-02/_temporary/1/_temporary/attempt_1528318855054_3528_m_00_1/part-m-0
> could only be replicated to 0 nodes instead of minReplication (=1).  There
> are 4 datanode(s) running and no node(s) are excluded in this operation.
>
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1735)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2561)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:829)
>
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510)
>
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
>
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
>
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
>
>
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1435)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1345)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>
> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:444)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
>
> at com.sun.proxy.$Proxy15.addBlock(Unknown Source)
>
> at
> org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1838)
>
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1638)
>
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704)
>
>
> References I already followed:
>
>
> https://community.hortonworks.com/articles/16144/write-or-append-failures-in-very-small-clusters-un.html
>
>
> https://stackoverflow.com/questions/14288453/writing-to-hdfs-from-java-getting-could-only-be-replicated-to-0-nodes-instead
>
> https://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo
>
>
> https://stackoverflow.com/questions/36015864/hadoop-be-replicated-to-0-nodes-instead-of-minreplication-1-there-are-1/36310025
>
>
> Any help is appreciated.
>
>
> Thanks
>
> Sowjanya
>

Re:

2018-06-13 Thread Thai Bui 
That error occurred usually because of disks nearly out of space. In your
EMR cluster, SSH into one of the nodes and do a `df -h` to check disk usage
in all of your EBS storages. HDFS is usually configured to be unhealthy
when disks it's writing to are >90% utilized. Once that happens, the
DataNode will just be taken out of the list of available nodes and in your
case, all the DataNode are not available, causing new blocks to be rejected
when the NameNode is requesting for a place to write to (0 available out of
4 nodes).

Even though your cluster said that there's 120Gb available, the available
space might not be where DataNode is configured to write to, thus the
misleading assumption that you still have available space. This also
happens when YARN and/or M/R logs are filling up the disks where the
DataNode is running.

On Wed, Jun 13, 2018 at 8:56 AM Sowjanya Kakarala 
wrote:

> Hi Sajid,
>
> As this is development environment, we have limited nodes (4datanodes
> 1masternode) on a unmanaged switch.
> So here each node will be treated as rack (managed by HDFS, which creates
> block copies) with one replica.
>
>
> On Wed, Jun 13, 2018 at 1:31 AM, Sajid Mohammed 
> wrote:
>
>> what is your rack topology ?
>>
>> On Tue, Jun 12, 2018 at 9:26 PM Sowjanya Kakarala 
>> wrote:
>>
>>> Hi Guys,
>>>
>>>
>>> I have 4datanodes and one master node EMR cluster with 120GB data
>>> storage left. I have been running sqoop jobs which loads data to hive
>>> table. After some jobs ran successfully I suddenly see these errors all
>>> over the name node logs and datanodes logs.
>>>
>>> I have tried changing so many configurations as suggeted in
>>> stackoverflow and hortonworks sites but couldnt find a way for fixing it.
>>>
>>>
>>> Here is the error:
>>>
>>> 2018-06-12 15:32:35,933 WARN [main] org.apache.hadoop.mapred.YarnChild:
>>> Exception running child : 
>>> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
>>> File
>>> /user/hive/warehouse/monolith.db/tblname/_SCRATCH0.28417629602676764/time_stamp=2018-04-02/_temporary/1/_temporary/attempt_1528318855054_3528_m_00_1/part-m-0
>>> could only be replicated to 0 nodes instead of minReplication (=1).  There
>>> are 4 datanode(s) running and no node(s) are excluded in this operation.
>>>
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1735)
>>>
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265)
>>>
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2561)
>>>
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:829)
>>>
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510)
>>>
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>>
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
>>>
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
>>>
>>> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
>>>
>>> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
>>>
>>>
>>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1435)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1345)
>>>
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
>>>
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>>
>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:444)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>
>>> at
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
>>>
>>> at
>>>

Re:

2018-06-13 Thread Thai Bui 
In which case, check your DataNode logs on one of the HDFS nodes. Also
check NameNode logs as well, the issue is related to HDFS not Hive so you
may have more luck debugging the problem there.

On Wed, Jun 13, 2018 at 11:16 AM Sowjanya Kakarala 
wrote:

> hmm, that is interesting. My df -h looks like below. I have all the logs
> and data in /mnt
>
> ~]$ df -h
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> devtmpfs 16G   56K   16G   1% /dev
>
> tmpfs16G 0   16G   0% /dev/shm
>
> /dev/nvme0n1p1  9.8G  6.1G  3.6G  63% /
>
> /dev/nvme1n1p1  5.0G  142M  4.9G   3% /emr
>
> /dev/nvme1n1p2  115G  2.2G  113G   2% /mnt
>
>
> On Wed, Jun 13, 2018 at 10:28 AM, Thai Bui  wrote:
>
>> That error occurred usually because of disks nearly out of space. In your
>> EMR cluster, SSH into one of the nodes and do a `df -h` to check disk usage
>> in all of your EBS storages. HDFS is usually configured to be unhealthy
>> when disks it's writing to are >90% utilized. Once that happens, the
>> DataNode will just be taken out of the list of available nodes and in your
>> case, all the DataNode are not available, causing new blocks to be rejected
>> when the NameNode is requesting for a place to write to (0 available out of
>> 4 nodes).
>>
>> Even though your cluster said that there's 120Gb available, the available
>> space might not be where DataNode is configured to write to, thus the
>> misleading assumption that you still have available space. This also
>> happens when YARN and/or M/R logs are filling up the disks where the
>> DataNode is running.
>>
>> On Wed, Jun 13, 2018 at 8:56 AM Sowjanya Kakarala 
>> wrote:
>>
>>> Hi Sajid,
>>>
>>> As this is development environment, we have limited nodes (4datanodes
>>> 1masternode) on a unmanaged switch.
>>> So here each node will be treated as rack (managed by HDFS, which
>>> creates block copies) with one replica.
>>>
>>>
>>> On Wed, Jun 13, 2018 at 1:31 AM, Sajid Mohammed 
>>> wrote:
>>>
 what is your rack topology ?

 On Tue, Jun 12, 2018 at 9:26 PM Sowjanya Kakarala 
 wrote:

> Hi Guys,
>
>
> I have 4datanodes and one master node EMR cluster with 120GB data
> storage left. I have been running sqoop jobs which loads data to hive
> table. After some jobs ran successfully I suddenly see these errors all
> over the name node logs and datanodes logs.
>
> I have tried changing so many configurations as suggeted in
> stackoverflow and hortonworks sites but couldnt find a way for fixing it.
>
>
> Here is the error:
>
> 2018-06-12 15:32:35,933 WARN [main]
> org.apache.hadoop.mapred.YarnChild: Exception running child :
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /user/hive/warehouse/monolith.db/tblname/_SCRATCH0.28417629602676764/time_stamp=2018-04-02/_temporary/1/_temporary/attempt_1528318855054_3528_m_00_1/part-m-0
> could only be replicated to 0 nodes instead of minReplication (=1).  There
> are 4 datanode(s) running and no node(s) are excluded in this operation.
>
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1735)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2561)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:829)
>
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510)
>
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
>
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
>
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
>
>
> at
> org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1435)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1345)
>
> at
>

Re:

2018-06-13 Thread Sowjanya Kakarala 
Hi Sajid,

As this is development environment, we have limited nodes (4datanodes
1masternode) on a unmanaged switch.
So here each node will be treated as rack (managed by HDFS, which creates
block copies) with one replica.


On Wed, Jun 13, 2018 at 1:31 AM, Sajid Mohammed 
wrote:

> what is your rack topology ?
>
> On Tue, Jun 12, 2018 at 9:26 PM Sowjanya Kakarala 
> wrote:
>
>> Hi Guys,
>>
>>
>> I have 4datanodes and one master node EMR cluster with 120GB data storage
>> left. I have been running sqoop jobs which loads data to hive table. After
>> some jobs ran successfully I suddenly see these errors all over the name
>> node logs and datanodes logs.
>>
>> I have tried changing so many configurations as suggeted in stackoverflow
>> and hortonworks sites but couldnt find a way for fixing it.
>>
>>
>> Here is the error:
>>
>> 2018-06-12 15:32:35,933 WARN [main] org.apache.hadoop.mapred.YarnChild:
>> Exception running child : 
>> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
>> File /user/hive/warehouse/monolith.db/tblname/_SCRATCH0.
>> 28417629602676764/time_stamp=2018-04-02/_temporary/1/_temporary/attempt_
>> 1528318855054_3528_m_00_1/part-m-0 could only be replicated to 0
>> nodes instead of minReplication (=1).  There are 4 datanode(s) running
>> and no node(s) are excluded in this operation.
>>
>> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.
>> chooseTarget4NewBlock(BlockManager.java:1735)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.
>> chooseTargetForNewBlock(FSDirWriteFileOp.java:265)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
>> getAdditionalBlock(FSNamesystem.java:2561)
>>
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
>> addBlock(NameNodeRpcServer.java:829)
>>
>> at org.apache.hadoop.hdfs.protocolPB.
>> ClientNamenodeProtocolServerSideTranslatorPB.addBlock(
>> ClientNamenodeProtocolServerSideTranslatorPB.java:510)
>>
>> at org.apache.hadoop.hdfs.protocol.proto.
>> ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
>> ClientNamenodeProtocolProtos.java)
>>
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
>> ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
>>
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
>>
>> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
>>
>> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>
>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>> UserGroupInformation.java:1836)
>>
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
>>
>>
>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1435)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1345)
>>
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
>> invoke(ProtobufRpcEngine.java:227)
>>
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
>> invoke(ProtobufRpcEngine.java:116)
>>
>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>
>> at org.apache.hadoop.hdfs.protocolPB.
>> ClientNamenodeProtocolTranslatorPB.addBlock(
>> ClientNamenodeProtocolTranslatorPB.java:444)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:62)
>>
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.
>> invokeMethod(RetryInvocationHandler.java:409)
>>
>> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
>> invokeMethod(RetryInvocationHandler.java:163)
>>
>> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
>> invoke(RetryInvocationHandler.java:155)
>>
>> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
>> invokeOnce(RetryInvocationHandler.java:95)
>>
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
>> RetryInvocationHandler.java:346)
>>
>> at com.sun.proxy.$Proxy15.addBlock(Unknown Source)
>>
>> at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(
>> DataStreamer.java:1838)
>>
>> at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(
>> DataStreamer.java:1638)
>>
>> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704)
>>
>>
>> References I already followed:
>>
>> https://community.hortonworks.com/articles/16144/write-or-
>> append-failures-in-very-small-clusters-un.html
>>
>> https://stackoverflow.com/questions/14288453/writing-to-
>>

Re:

2018-06-13 Thread Sowjanya Kakarala 
hmm, that is interesting. My df -h looks like below. I have all the logs
and data in /mnt

~]$ df -h

Filesystem  Size  Used Avail Use% Mounted on

devtmpfs 16G   56K   16G   1% /dev

tmpfs16G 0   16G   0% /dev/shm

/dev/nvme0n1p1  9.8G  6.1G  3.6G  63% /

/dev/nvme1n1p1  5.0G  142M  4.9G   3% /emr

/dev/nvme1n1p2  115G  2.2G  113G   2% /mnt


On Wed, Jun 13, 2018 at 10:28 AM, Thai Bui  wrote:

> That error occurred usually because of disks nearly out of space. In your
> EMR cluster, SSH into one of the nodes and do a `df -h` to check disk usage
> in all of your EBS storages. HDFS is usually configured to be unhealthy
> when disks it's writing to are >90% utilized. Once that happens, the
> DataNode will just be taken out of the list of available nodes and in your
> case, all the DataNode are not available, causing new blocks to be rejected
> when the NameNode is requesting for a place to write to (0 available out of
> 4 nodes).
>
> Even though your cluster said that there's 120Gb available, the available
> space might not be where DataNode is configured to write to, thus the
> misleading assumption that you still have available space. This also
> happens when YARN and/or M/R logs are filling up the disks where the
> DataNode is running.
>
> On Wed, Jun 13, 2018 at 8:56 AM Sowjanya Kakarala 
> wrote:
>
>> Hi Sajid,
>>
>> As this is development environment, we have limited nodes (4datanodes
>> 1masternode) on a unmanaged switch.
>> So here each node will be treated as rack (managed by HDFS, which creates
>> block copies) with one replica.
>>
>>
>> On Wed, Jun 13, 2018 at 1:31 AM, Sajid Mohammed 
>> wrote:
>>
>>> what is your rack topology ?
>>>
>>> On Tue, Jun 12, 2018 at 9:26 PM Sowjanya Kakarala 
>>> wrote:
>>>
 Hi Guys,


 I have 4datanodes and one master node EMR cluster with 120GB data
 storage left. I have been running sqoop jobs which loads data to hive
 table. After some jobs ran successfully I suddenly see these errors all
 over the name node logs and datanodes logs.

 I have tried changing so many configurations as suggeted in
 stackoverflow and hortonworks sites but couldnt find a way for fixing it.


 Here is the error:

 2018-06-12 15:32:35,933 WARN [main] org.apache.hadoop.mapred.YarnChild:
 Exception running child : org.apache.hadoop.ipc.RemoteException(java.io
 .IOException): File /user/hive/warehouse/monolith.db/tblname/_SCRATCH0.
 28417629602676764/time_stamp=2018-04-02/_temporary/1/_
 temporary/attempt_1528318855054_3528_m_00_1/part-m-0 could
 only be replicated to 0 nodes instead of minReplication (=1).  There
 are 4 datanode(s) running and no node(s) are excluded in this operation.

 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.
 chooseTarget4NewBlock(BlockManager.java:1735)

 at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.
 chooseTargetForNewBlock(FSDirWriteFileOp.java:265)

 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
 getAdditionalBlock(FSNamesystem.java:2561)

 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
 addBlock(NameNodeRpcServer.java:829)

 at org.apache.hadoop.hdfs.protocolPB.
 ClientNamenodeProtocolServerSideTranslatorPB.addBlock(
 ClientNamenodeProtocolServerSideTranslatorPB.java:510)

 at org.apache.hadoop.hdfs.protocol.proto.
 ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.
 callBlockingMethod(ClientNamenodeProtocolProtos.java)

 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
 ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)

 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)

 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)

 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)

 at java.security.AccessController.doPrivileged(Native Method)

 at javax.security.auth.Subject.doAs(Subject.java:422)

 at org.apache.hadoop.security.UserGroupInformation.doAs(
 UserGroupInformation.java:1836)

 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)


 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:
 1489)

 at org.apache.hadoop.ipc.Client.call(Client.java:1435)

 at org.apache.hadoop.ipc.Client.call(Client.java:1345)

 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
 invoke(ProtobufRpcEngine.java:227)

 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
 invoke(ProtobufRpcEngine.java:116)

 at com.sun.proxy.$Proxy14.addBlock(Unknown Source)

 at org.apache.hadoop.hdfs.protocolPB.
 ClientNamenodeProtocolTranslatorPB.addBlock(

hive set block size not working

2018-06-13 Thread cathy zhu 
I wanted to find the optimized parquet file size. It looks like no matther
how much I put on set block size, hive always gave the same result on
parquet file sizes.

I was copying everything from a table to another same dummy table for the
experiment. There are a lot small files. Here are the table properties. Can
anyone help me? Thanks in advance!



SET hive.exec.dynamic.partition.mode=nonstrict;

SET parquet.column.index.access=true;

SET hive.merge.mapredfiles=true;

SET hive.exec.compress.output=true;

SET
mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;

SET mapred.output.compression.type=BLOCK;

SET parquet.compression=SNAPPY;

SET dfs.block.size=445644800;

SET parquet.block.size=445644800;

Hive 3,0 on Hadoop 3.0.3 crahes with org.apache.hadoop.mapreduce.v2.app.MRAppMaster error

2018-06-13 Thread Mich Talebzadeh 
*Hadoop 3.0.3Hive (version 3.0.0)*

Running a simple query

select count(1) from sales;

I get the following error in container

Error: Could not find or load main class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster

The container file launch_container.sh has the following entry

exec /bin/bash -c "$JAVA_HOME/bin/java
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/home/hduser/hadoop-3.0.3/logs/userlogs/application_1528926568884_0006/container_1528926568884_0006_02_01


-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
-Dhadoop.root.logfile=syslog

 -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster

1>/home/hduser/hadoop-3.0.3/logs/userlogs/application_1528926568884_0006/container_1528926568884_0006_02_01/stdout
2>/home/hduser/hadoop-3.0.3/logs/userlogs/application_1528926568884_0006/container_1528926568884_0006_02_01/stderr
"

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: Hive 3,0 on Hadoop 3.0.3 crahes with org.apache.hadoop.mapreduce.v2.app.MRAppMaster error

2018-06-13 Thread Mich Talebzadeh 
sorted out. needed to add the following to
${HADOOP_HOME}/etc/Hadoop/mapred-site.xml !

   
mapreduce.framework.name
yarn


yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}


mapreduce.map.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}


mapreduce.reduce.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 13 June 2018 at 23:49, Mich Talebzadeh  wrote:

>
> *Hadoop 3.0.3Hive (version 3.0.0)*
>
> Running a simple query
>
> select count(1) from sales;
>
> I get the following error in container
>
> Error: Could not find or load main class org.apache.hadoop.mapreduce.
> v2.app.MRAppMaster
>
> The container file launch_container.sh has the following entry
>
> exec /bin/bash -c "$JAVA_HOME/bin/java 
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hduser/hadoop-3.0.3/logs/
> userlogs/application_1528926568884_0006/container_1528926568884_0006_02_01
>
>
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog
>
>  -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>
> 1>/home/hduser/hadoop-3.0.3/logs/userlogs/application_
> 1528926568884_0006/container_1528926568884_0006_02_01/stdout
> 2>/home/hduser/hadoop-3.0.3/logs/userlogs/application_
> 1528926568884_0006/container_1528926568884_0006_02_01/stderr "
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>

How to unsubscribe

2018-06-13 Thread Lefty Leverenz 
Many people try to unsubscribe from this mailing list by sending a message
directly to the list.  That doesn't work.

To unsubscribe, you have to send a message (any message) to the automated
unsubscribe address:

user-unsubscr...@hive.apache.org


as described here:  Hive Mailing Lists
.


Since it's automated, the message must be sent using the email account that
is subscribed to the Hive user mailing list.

-- Lefty