Just use setParallelism(). This specifies how many threads are used for the
operator.
writeAsText("file:///tmp/res.txt").setParallelism(1);
This will give you a single output file.
Cheers,
Robert
On Wed, Oct 29, 2014 at 10:22 PM, Flavio Pompermaier <[email protected]>
wrote:
> Would it be that difficult to change the behaviour for file:/// and create
> a single file?or is there a way to do that?
> On Oct 29, 2014 9:52 PM, "Márton Balassi" <[email protected]>
> wrote:
>
>> Dear Flavio,
>>
>> Yes, the writeAsText() merthod really creates a folder which contains a
>> file for each execution thread, so your threads do not block each other and
>> the execution can use multiple cores on your machine. You can see similar
>> results if you try it with env.execute() from an IDE.
>>
>> There are filesystems, HDFS to mention the most prominent one which can
>> transparently treat such folder structure as a single file and then it
>> would behave as you expect. I hope this answers your question.
>>
>> Best,
>>
>> Marton
>>
>> On Wed, Oct 29, 2014 at 8:31 PM, Flavio Pompermaier <[email protected]
>> > wrote:
>>
>>> Hi to all,
>>> running the example at
>>> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
>>> I was thinking that the writeAsText on a local file was creating a text
>>> file on my local filesystem..instead it creates something similar to a
>>> sequence file (within a folder).
>>> This is something misleading I think...or the API name is wrong or this
>>> is a bug (IMHO).
>>> Btw..how can I modify the following program to write results in a single
>>> text file on my local filesystem?
>>>
>>> public static void main(String[] args) throws Exception {
>>> ExecutionEnvironment env = ExecutionEnvironment.createLocalEnvironment();
>>> DataSet<String> data = env.readTextFile("file:///tmp/res.txt");
>>> data.filter(new FilterFunction<String>() {
>>> public boolean filter(String value) {
>>> return value.startsWith("http://";);
>>> }
>>> }).writeAsText("file:///tmp/res.txt");
>>> env.execute();}
>>>
>>> Best,
>>> Flavio
>>>
>>>
>>>
>>
