Yep, it's on its plans to move to more memory. But i don't think this is what is causing the issue here. Le 2 mai 2013 17:12, "Andrew Purtell" <[email protected]> a écrit :
> > OS is Ubuntu 12.04 and instance type is c1.medium > > Eeek! > > You shouldn't use less than c1.xlarge for running Hadoop+HBase on EC2. A > c1.medium has only 7 GB of RAM in total. > > > On Thu, May 2, 2013 at 1:53 PM, Loic Talon <[email protected]> wrote: > > > Hi Andrew, > > Thanks for those responses. > > > > The server has been deployed by Cloudera Manager. > > OS is Ubuntu 12.04 and instance type is c1.medium. > > Instance store are used, not EBS. > > > > It's possible that this problem is a memory problem ? > > Because when region server hab been started I have in stdout.log : > > > > Thu May 2 17:01:10 UTC 2013 > > using /usr/lib/jvm/j2sdk1.6-oracle as JAVA_HOME > > using 4 as CDH_VERSION > > using as HBASE_HOME > > using /run/cloudera-scm-agent/process/381-hbase-REGIONSERVER as > > HBASE_CONF_DIR > > using /run/cloudera-scm-agent/process/381-hbase-REGIONSERVER as > > HADOOP_CONF_DIR > > using as HADOOP_HOME > > > > But when I have the problem, I have in stdout.log : > > Thu May 2 17:01:10 UTC 2013 > > using /usr/lib/jvm/j2sdk1.6-oracle as JAVA_HOME > > using 4 as CDH_VERSION > > using as HBASE_HOME > > using /run/cloudera-scm-agent/process/381-hbase-REGIONSERVER as > > HBASE_CONF_DIR > > using /run/cloudera-scm-agent/process/381-hbase-REGIONSERVER as > > HADOOP_CONF_DIR > > using as HADOOP_HOME > > # > > # java.lang.OutOfMemoryError: Java heap space > > # -XX:OnOutOfMemoryError="kill -9 %p" > > # Executing /bin/sh -c "kill -9 20140"... > > > > Thanks > > > > Loic > > > > > > > > > > > > > > Loïc TALON > > > > > > [email protected] <http://teads.tv/> > > Video Ads Solutions > > > > > > > > 2013/5/2 Andrew Purtell <[email protected]> > > > > > Every instance type except t1.micro has a certain number of instance > > > storage (locally attached disk) volumes available, 1, 2, or 4 depending > > on > > > type. > > > > > > As you probably know, you can use or create AMIs backed by > > instance-store, > > > in which the OS image is constructed on locally attached disk by a > > parallel > > > fetch process from slices of the root volume image stored in S3, or > > backed > > > by EBS, in which case the OS image is an EBS volume and attached over > the > > > network, like a SAN. > > > > > > If you launch an Amazon Linux instance store backed instance the first > > > "ephemeral" local volume will be automatically attached on > > > /media/ephemeral0. That's where that term comes from, it's a synonym > for > > > instance-store. (You can by the way tell CloudInit via directives sent > > over > > > instance data to mount all of them.) > > > > > > If you have an EBS backed instance the default is to NOT attach any of > > > these volumes. > > > > > > If you are launching your instance with the Amazon Web console, in the > > > volume configuration part you can set up instance-store aka "ephemeral" > > > mounts whether it is instance-store backed or EBS backed. > > > > > > Sorry I can't get into more background on this. Hope it helps. > > > > > > > > > > > > On Thu, May 2, 2013 at 1:23 PM, Jean-Marc Spaggiari < > > > [email protected] > > > > wrote: > > > > > > > Hi Andrew, > > > > > > > > No, this AWS instance is configured with instance stores too. > > > > > > > > What do you mean by "ephemeral"? > > > > > > > > JM > > > > > > > > 2013/5/2 Andrew Purtell <[email protected]> > > > > > > > > > Oh, I have faced issues with Hadoop on AWS personally. :-) But not > > this > > > > > one. I use instance-store aka "ephemeral" volumes for DataNode > block > > > > > storage. Are you by chance using EBS? > > > > > > > > > > > > > > > On Thu, May 2, 2013 at 1:10 PM, Jean-Marc Spaggiari < > > > > > [email protected] > > > > > > wrote: > > > > > > > > > > > But that's wierld. This instance is running on AWS. If there > issues > > > > with > > > > > > Hadoop and AWS I think some other people will have faced it > before > > > me. > > > > > > > > > > > > Ok. I will move the discussion on the Hadoop mailing list since > it > > > > seems > > > > > to > > > > > > be more related to hadoop vs OS. > > > > > > > > > > > > Thank, > > > > > > > > > > > > JM > > > > > > > > > > > > 2013/5/2 Andrew Purtell <[email protected]> > > > > > > > > > > > > > > 2013-05-02 14:02:41,063 INFO > org.apache.hadoop.hdfs.DFSClient: > > > > > > Exception > > > > > > > in > > > > > > > createBlockOutputStream java.io.EOFException: Premature EOF: no > > > > length > > > > > > > prefix available > > > > > > > > > > > > > > The DataNode aborted the block transfer. > > > > > > > > > > > > > > > 2013-05-02 14:02:41,063 ERROR org.apache.hadoop.hdfs.server. > > > > > > > datanode.DataNode: > > > > > > > ip-10-238-38-193.eu-west-1.compute.internal:50010:DataXceiver > > > > > > > error processing WRITE_BLOCK operation src: / > > 10.238.38.193:39831 > > > > dest: > > > > > > / > > > > > > > 10.238.38.193:50010 java.io.FileNotFoundException: > > > > > > /mnt/dfs/dn/current/BP- > > > > > > > 1179773663-10.238.38.193-1363960970263/current/rbw/blk_ > > > > > > > 7082931589039745816_1955950.meta (Invalid argument) > > > > > > > > at java.io.RandomAccessFile.open(Native Method) > > > > > > > > at > > > > java.io.RandomAccessFile.<init>(RandomAccessFile.java:216) > > > > > > > > > > > > > > This looks like the native (OS level) side of RAF got EINVAL > back > > > > from > > > > > > > create() or open(). Go from there. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 2, 2013 at 12:27 PM, Jean-Marc Spaggiari < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Any idea what can be the cause of a "Premature EOF: no length > > > > prefix > > > > > > > > available" error? > > > > > > > > > > > > > > > > 2013-05-02 14:02:41,063 INFO > org.apache.hadoop.hdfs.DFSClient: > > > > > > Exception > > > > > > > in > > > > > > > > createBlockOutputStream > > > > > > > > java.io.EOFException: Premature EOF: no length prefix > available > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:171) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1105) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1039) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) > > > > > > > > 2013-05-02 14:02:41,064 INFO > org.apache.hadoop.hdfs.DFSClient: > > > > > > Abandoning > > > > > > > > > > > > > > > > > > > > BP-1179773663-10.238.38.193-1363960970263:blk_7082931589039745816_1955950 > > > > > > > > 2013-05-02 14:02:41,068 INFO > org.apache.hadoop.hdfs.DFSClient: > > > > > > Excluding > > > > > > > > datanode 10.238.38.193:50010 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm getting that on a server start. Logs are splitted > > correctly, > > > > > > > > coprocessors deployed corretly, and then I'm getting this > > > > exception. > > > > > > It's > > > > > > > > excluding the datanode, and because of that almost everything > > > > > remaining > > > > > > > is > > > > > > > > failing. > > > > > > > > > > > > > > > > There is only one server in this "cluster"... But even so, it > > > > should > > > > > be > > > > > > > > working. There is one master, one RS, one NN and one DN. On a > > AWS > > > > > host. > > > > > > > > > > > > > > > > At the same time on the hadoop datanode side I'm getting > that: > > > > > > > > > > > > > > > > 2013-05-02 14:02:41,063 INFO > > > > > > > > org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock > > > > > > > > > > > > > > > > > > > > BP-1179773663-10.238.38.193-1363960970263:blk_7082931589039745816_1955950 > > > > > > > > received exception java.io.FileNotFoundException: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /mnt/dfs/dn/current/BP-1179773663-10.238.38.193-1363960970263/current/rbw/blk_7082931589039745816_1955950.meta > > > > > > > > (Invalid argument) > > > > > > > > 2013-05-02 14:02:41,063 ERROR > > > > > > > > org.apache.hadoop.hdfs.server.datanode.DataNode: > > > > > > > > ip-10-238-38-193.eu-west-1.compute.internal:50010:DataXceiver > > > error > > > > > > > > processing WRITE_BLOCK operation src: /10.238.38.193:39831 > > dest: > > > > / > > > > > > > > 10.238.38.193:50010 > > > > > > > > java.io.FileNotFoundException: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /mnt/dfs/dn/current/BP-1179773663-10.238.38.193-1363960970263/current/rbw/blk_7082931589039745816_1955950.meta > > > > > > > > (Invalid argument) > > > > > > > > at java.io.RandomAccessFile.open(Native Method) > > > > > > > > at > > > > java.io.RandomAccessFile.<init>(RandomAccessFile.java:216) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.createStreams(ReplicaInPipeline.java:187) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:199) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:457) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:103) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:67) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221) > > > > > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > > > > > > > > > > > > > > > > > Does is sound more an hadoop issue than an HBase one? > > > > > > > > > > > > > > > > JM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Best regards, > > > > > > > > > > > > > > - Andy > > > > > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - > > Piet > > > > > Hein > > > > > > > (via Tom White) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best regards, > > > > > > > > > > - Andy > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > > Hein > > > > > (via Tom White) > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > > > > - Andy > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein > > > (via Tom White) > > > > > > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) >
