Hey Amnon, So just to make sure I understand - you also saw the same issue with 1.0.2? Just asking because whether or not this regresses the 1.0.2 behavior is important for our own bug tracking.
- Patrick On Mon, Aug 25, 2014 at 10:22 PM, Amnon Khen <amnon...@gmail.com> wrote: > There were no failures nor exceptions. > > > On Tue, Aug 26, 2014 at 1:31 AM, Matei Zaharia <matei.zaha...@gmail.com> > wrote: > > > Got it. Another thing that would help is if you spot any exceptions or > > failed tasks in the web UI (http://<driver>:4040). > > > > Matei > > > > On August 25, 2014 at 3:07:41 PM, amnonkhen (amnon...@gmail.com) wrote: > > > > Hi Matei, > > The original issue happened on a spark-1.0.2-bin-hadoop2 installation. > > I will try the synthetic operation and see if I get the same results or > > not. > > Amnon > > > > > > On Mon, Aug 25, 2014 at 11:26 PM, Matei Zaharia [via Apache Spark > > Developers List] <ml-node+s1001551n8000...@n3.nabble.com> wrote: > > > > > Was the original issue with Spark 1.1 (i.e. master branch) or an > earlier > > > release? > > > > > > One possibility is that your S3 bucket is in a remote Amazon region, > > which > > > would make it very slow. In my experience though saveAsTextFile has > > worked > > > even for pretty large datasets in that situation, so maybe there's > > > something else in your job causing a problem. Have you tried other > > > operations on the data, like count(), or saving synthetic datasets > (e.g. > > > sc.parallelize(1 to 100*1000*1000, 20).saveAsTextFile(...)? > > > > > > Matei > > > > > > On August 25, 2014 at 12:09:25 PM, amnonkhen ([hidden email] > > > <http://user/SendEmail.jtp?type=node&node=8000&i=0>) wrote: > > > > > > Hi jerryye, > > > Maybe if you voted up my question on Stack Overflow it would get some > > > traction and we would get nearer to a solution. > > > Thanks, > > > Amnon > > > > > > > > > > > > -- > > > View this message in context: > > > > > > http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spark-does-not-work-just-hangs-tp7795p7991.html > > > > > > Sent from the Apache Spark Developers List mailing list archive at > > > Nabble.com. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [hidden email] > > > <http://user/SendEmail.jtp?type=node&node=8000&i=1> > > > For additional commands, e-mail: [hidden email] > > > <http://user/SendEmail.jtp?type=node&node=8000&i=2> > > > > > > > > > > > > ------------------------------ > > > If you reply to this email, your message will be added to the > discussion > > > below: > > > > > > > > > http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spark-does-not-work-just-hangs-tp7795p8000.html > > > To unsubscribe from saveAsTextFile to s3 on spark does not work, just > > > hangs, click here > > > < > > > http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7795&code=YW1ub24uaXNAZ21haWwuY29tfDc3OTV8LTkxODIwMjYzNg== > > > > > > > . > > > NAML > > > < > > > http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml > > > > > > > > > > > > > > > > > -- > > View this message in context: > > > http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spark-does-not-work-just-hangs-tp7795p8008.html > > Sent from the Apache Spark Developers List mailing list archive at > > Nabble.com. > > > > >
