Re: Is anyone using Amazon EC2? (second attempt!)
Hi , Any update on this? I am not sure if the issue I am seeing is related .. I have 8 slaves and when I created the cluster I specified ebs volume with 100G. I see on Ec2 8 volumes created and each attached to the corresponding slave. But when I try to copy data on it , it complains that /root/ephemeral-hdfs/bin/hadoop fs -cp /intersection hdfs:// ec2-54-149-112-136.us-west-2.compute.amazonaws.com:9010/ 2015-05-28 23:40:35,447 WARN hdfs.DFSClient (DFSOutputStream.java:run(562)) - DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /intersection/kmer150/commonGoodKmers/_temporary/_attempt_201504010056_0004_m_000428_3948/part-00428._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation. It shows only 1 datanode , but for ephermal-hdfs it shows 8 datanodes. Any thoughts? Thanks _R On Sat, May 23, 2015 at 7:24 AM, Joe Wass jw...@crossref.org wrote: I used Spark on EC2 a while ago, but recent revisions seem to have broken the functionality. Is anyone actually using Spark on EC2 at the moment? The bug in question is: https://issues.apache.org/jira/browse/SPARK-5008 It makes it impossible to use persistent HDFS without a workround on each slave node. No-one seems to be interested in the bug, so I wonder if other people aren't actually having this problem. If this is the case, any suggestions? Joe
Re: Is anyone using Amazon EC2? (second attempt!)
I use spark on EC2 but it's a CDH 5.3.3 distribution (starving developer version) installed thru Cloudera Manager. Spark is configured to run on Yarn. Regards Sanjay Sent from my iPhone On May 29, 2015, at 6:16 PM, roni roni.epi...@gmail.com wrote: Hi , Any update on this? I am not sure if the issue I am seeing is related .. I have 8 slaves and when I created the cluster I specified ebs volume with 100G. I see on Ec2 8 volumes created and each attached to the corresponding slave. But when I try to copy data on it , it complains that /root/ephemeral-hdfs/bin/hadoop fs -cp /intersection hdfs://ec2-54-149-112-136.us-west-2.compute.amazonaws.com:9010/ 2015-05-28 23:40:35,447 WARN hdfs.DFSClient (DFSOutputStream.java:run(562)) - DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /intersection/kmer150/commonGoodKmers/_temporary/_attempt_201504010056_0004_m_000428_3948/part-00428._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation. It shows only 1 datanode , but for ephermal-hdfs it shows 8 datanodes. Any thoughts? Thanks _R On Sat, May 23, 2015 at 7:24 AM, Joe Wass jw...@crossref.org wrote: I used Spark on EC2 a while ago, but recent revisions seem to have broken the functionality. Is anyone actually using Spark on EC2 at the moment? The bug in question is: https://issues.apache.org/jira/browse/SPARK-5008 It makes it impossible to use persistent HDFS without a workround on each slave node. No-one seems to be interested in the bug, so I wonder if other people aren't actually having this problem. If this is the case, any suggestions? Joe
Re: Is anyone using Amazon EC2?
Yes, we're running Spark on EC2. Will transition to EMR soon. -Vadim ᐧ On Sat, May 23, 2015 at 2:22 PM, Johan Beisser j...@caustic.org wrote: Yes. We're looking at bootstrapping in EMR... On Sat, May 23, 2015 at 07:21 Joe Wass jw...@crossref.org wrote: I used Spark on EC2 a while ago
Re: Is anyone using Amazon EC2?
Sorry guys, my email submitted before I finished writing it. Check my other message (with the same subject)! On 23 May 2015 at 20:25, Shafaq s.abdullah...@gmail.com wrote: Yes-Spark EC2 cluster . Looking into migrating to spark emr. Adding more ec2 is not possible afaik. On May 23, 2015 11:22 AM, Johan Beisser j...@caustic.org wrote: Yes. We're looking at bootstrapping in EMR... On Sat, May 23, 2015 at 07:21 Joe Wass jw...@crossref.org wrote: I used Spark on EC2 a while ago
Is anyone using Amazon EC2?
I used Spark on EC2 a while ago
Re: Is anyone using Amazon EC2?
Yes. We're looking at bootstrapping in EMR... On Sat, May 23, 2015 at 07:21 Joe Wass jw...@crossref.org wrote: I used Spark on EC2 a while ago
Re: Is anyone using Amazon EC2?
Yes-Spark EC2 cluster . Looking into migrating to spark emr. Adding more ec2 is not possible afaik. On May 23, 2015 11:22 AM, Johan Beisser j...@caustic.org wrote: Yes. We're looking at bootstrapping in EMR... On Sat, May 23, 2015 at 07:21 Joe Wass jw...@crossref.org wrote: I used Spark on EC2 a while ago
