Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-07 Thread DEEP NARAYAN Singh
Hi Wei and Till, Thanks for the quick reply. *@Wei,* I tried with code which you have suggested and it is working fine but I have one use case where it is failing, below is the csv input data format : Csv file data format : --- *field_id,data,* *A,1B,3C,4D,9*

Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-07 Thread Till Rohrmann
Hi Deep, Could you use the TextInputFormat which reads a file line by line? That way you can do the JSON parsing as part of a mapper which consumes the file lines. Cheers, Till On Mon, Dec 7, 2020 at 1:05 PM Wei Zhong wrote: > Hi Deep, > > (redirecting this to user mailing list as this is not

Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-07 Thread Wei Zhong
Hi Deep, (redirecting this to user mailing list as this is not a dev question) You can try to set the line delimiter and field delimiter of the RowCsvInputFormat to a non-printing character (assume there is no non-printing characters in the csv files). It will read all the content of a csv

Urgent help on S3 CSV file reader DataStream Job

2020-12-07 Thread DEEP NARAYAN Singh
Hi Guys, Below is my code snippet , which read all csv files under the given folder row by row but my requirement is to read csv file at a time and convert as json which will looks like : {"A":"1","B":"3","C":"4","D":9} Csv file data format : --- *field_id,data,*