Re:

2018-06-13 Thread Sajid Mohammed
what is your rack topology ? On Tue, Jun 12, 2018 at 9:26 PM Sowjanya Kakarala wrote: > Hi Guys, > > > I have 4datanodes and one master node EMR cluster with 120GB data storage > left. I have been running sqoop jobs which loads data to hive table. After > some jobs ran successfully I suddenly

Re:

2018-06-13 Thread Thai Bui
That error occurred usually because of disks nearly out of space. In your EMR cluster, SSH into one of the nodes and do a `df -h` to check disk usage in all of your EBS storages. HDFS is usually configured to be unhealthy when disks it's writing to are >90% utilized. Once that happens, the

Re:

2018-06-13 Thread Thai Bui
In which case, check your DataNode logs on one of the HDFS nodes. Also check NameNode logs as well, the issue is related to HDFS not Hive so you may have more luck debugging the problem there. On Wed, Jun 13, 2018 at 11:16 AM Sowjanya Kakarala wrote: > hmm, that is interesting. My df -h looks

Re:

2018-06-13 Thread Sowjanya Kakarala
Hi Sajid, As this is development environment, we have limited nodes (4datanodes 1masternode) on a unmanaged switch. So here each node will be treated as rack (managed by HDFS, which creates block copies) with one replica. On Wed, Jun 13, 2018 at 1:31 AM, Sajid Mohammed wrote: > what is your

Re:

2018-06-13 Thread Sowjanya Kakarala
hmm, that is interesting. My df -h looks like below. I have all the logs and data in /mnt ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 16G 56K 16G 1% /dev tmpfs16G 0 16G 0% /dev/shm /dev/nvme0n1p1 9.8G 6.1G 3.6G 63% /

hive set block size not working

2018-06-13 Thread cathy zhu
I wanted to find the optimized parquet file size. It looks like no matther how much I put on set block size, hive always gave the same result on parquet file sizes. I was copying everything from a table to another same dummy table for the experiment. There are a lot small files. Here are the

Hive 3,0 on Hadoop 3.0.3 crahes with org.apache.hadoop.mapreduce.v2.app.MRAppMaster error

2018-06-13 Thread Mich Talebzadeh
*Hadoop 3.0.3Hive (version 3.0.0)* Running a simple query select count(1) from sales; I get the following error in container Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster The container file launch_container.sh has the following entry exec /bin/bash

Re: Hive 3,0 on Hadoop 3.0.3 crahes with org.apache.hadoop.mapreduce.v2.app.MRAppMaster error

2018-06-13 Thread Mich Talebzadeh
sorted out. needed to add the following to ${HADOOP_HOME}/etc/Hadoop/mapred-site.xml ! mapreduce.framework.name yarn yarn.app.mapreduce.am.env HADOOP_MAPRED_HOME=${HADOOP_HOME} mapreduce.map.env

How to unsubscribe

2018-06-13 Thread Lefty Leverenz
Many people try to unsubscribe from this mailing list by sending a message directly to the list. That doesn't work. To unsubscribe, you have to send a message (any message) to the automated unsubscribe address: user-unsubscr...@hive.apache.org as described here: Hive Mailing Lists