Nop Sir, it is possible - check my reply earlier -----Original Message----- From: Sean Owen [mailto:[email protected]] Sent: Thursday, April 16, 2015 6:35 PM To: Vadim Bichutskiy Cc: [email protected] Subject: Re: saveAsTextFile
You can't, since that's how it's designed to work. Batches are saved in different "files", which are really directories containing partitions, as is common in Hadoop. You can move them later, or just read them where they are. On Thu, Apr 16, 2015 at 6:32 PM, Vadim Bichutskiy <[email protected]> wrote: > I am using Spark Streaming where during each micro-batch I output data > to S3 using saveAsTextFile. Right now each batch of data is put into > its own directory containing > 2 objects, "_SUCCESS" and "part-00000." > > How do I output each batch into a common directory? > > Thanks, > Vadim > ᐧ --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
