Hi Timo
FileInputFormat fileInputFormat = new TextInputFormat(new
Path(fileSystem+this.path));
fileInputFormat.setNestedFileEnumeration(true);
fileInputFormat.setFilesFilter(new UnicaFileFilter(".csv"));
DataStream<String>value =this.execEnv.readFile(fileInputFormat,
fileSystem+this.path,
FileProcessingMode.PROCESS_CONTINUOUSLY, 2L).setParrallelism(1);
*1) Now if i set parallelism equals to 1 the file get sequentially
processed .*
*2) Modify splits **are being processed on the same task manger
sequentially.*
*3) I want to move the files after being processed.(How to achieve this) .*
On Mon, Mar 26, 2018 at 3:27 PM, Timo Walther <twal...@apache.org> wrote:
> Hi Puneet,
>
> can you share a little code example with us? I could not reproduce your
> problem.
>
> You have to keep in mind that a setParallelism() only affects the last
> operation. If you want to change the default parallelism of the entire
> pipeline, you have to change it in StreamExecutionEnvironment. Otherwise
> every following operator will again have the full parallelism which leads
> to a shuffle operation after your source.
>
> I hope this helps.
>
> Regards,
> Timo
>
>
> Am 22.03.18 um 09:17 schrieb Puneet Kinra:
>
> if i set parallelsim equals to 1 still it create multiple splits while
> processing.
>
> --
> *Cheers *
>
> *Puneet Kinra*
>
> *Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com
> <puneet.ki...@customercentria.com>*
>
> *e-mail :puneet.ki...@customercentria.com
> <puneet.ki...@customercentria.com>*
>
>
>
>
--
*Cheers *
*Puneet Kinra*
*Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com
<puneet.ki...@customercentria.com>*
*e-mail :puneet.ki...@customercentria.com
<puneet.ki...@customercentria.com>*
