Nop Sir, it is possible - check my reply earlier -----Original Message----- From: Sean Owen [mailto:so...@cloudera.com] Sent: Thursday, April 16, 2015 6:35 PM To: Vadim Bichutskiy Cc: user@spark.apache.org Subject: Re: saveAsTextFile
You can't, since that's how it's designed to work. Batches are saved in different "files", which are really directories containing partitions, as is common in Hadoop. You can move them later, or just read them where they are. On Thu, Apr 16, 2015 at 6:32 PM, Vadim Bichutskiy <vadim.bichuts...@gmail.com> wrote: > I am using Spark Streaming where during each micro-batch I output data > to S3 using saveAsTextFile. Right now each batch of data is put into > its own directory containing > 2 objects, "_SUCCESS" and "part-00000." > > How do I output each batch into a common directory? > > Thanks, > Vadim > ᐧ --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
