what is your rack topology ?
On Tue, Jun 12, 2018 at 9:26 PM Sowjanya Kakarala
wrote:
> Hi Guys,
>
>
> I have 4datanodes and one master node EMR cluster with 120GB data storage
> left. I have been running sqoop jobs which loads data to hive table. After
> some jobs ran successfully I suddenly
That error occurred usually because of disks nearly out of space. In your
EMR cluster, SSH into one of the nodes and do a `df -h` to check disk usage
in all of your EBS storages. HDFS is usually configured to be unhealthy
when disks it's writing to are >90% utilized. Once that happens, the
In which case, check your DataNode logs on one of the HDFS nodes. Also
check NameNode logs as well, the issue is related to HDFS not Hive so you
may have more luck debugging the problem there.
On Wed, Jun 13, 2018 at 11:16 AM Sowjanya Kakarala
wrote:
> hmm, that is interesting. My df -h looks
Hi Sajid,
As this is development environment, we have limited nodes (4datanodes
1masternode) on a unmanaged switch.
So here each node will be treated as rack (managed by HDFS, which creates
block copies) with one replica.
On Wed, Jun 13, 2018 at 1:31 AM, Sajid Mohammed
wrote:
> what is your
hmm, that is interesting. My df -h looks like below. I have all the logs
and data in /mnt
~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 56K 16G 1% /dev
tmpfs16G 0 16G 0% /dev/shm
/dev/nvme0n1p1 9.8G 6.1G 3.6G 63% /
I wanted to find the optimized parquet file size. It looks like no matther
how much I put on set block size, hive always gave the same result on
parquet file sizes.
I was copying everything from a table to another same dummy table for the
experiment. There are a lot small files. Here are the
*Hadoop 3.0.3Hive (version 3.0.0)*
Running a simple query
select count(1) from sales;
I get the following error in container
Error: Could not find or load main class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
The container file launch_container.sh has the following entry
exec /bin/bash
sorted out. needed to add the following to
${HADOOP_HOME}/etc/Hadoop/mapred-site.xml !
mapreduce.framework.name
yarn
yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}
mapreduce.map.env
Many people try to unsubscribe from this mailing list by sending a message
directly to the list. That doesn't work.
To unsubscribe, you have to send a message (any message) to the automated
unsubscribe address:
user-unsubscr...@hive.apache.org
as described here: Hive Mailing Lists