You can use csv reader with delimiter as '|' to split data and create a dataframe on top of the file data. Second step, filter the dataframe on column value like indicator type=A,D etc and put it in tables. Saving to tables you can use dataframewriter(not sure what is you destination db type here).
You can keep this part where based on column type you have to filter data generic and read the configurations like say - col value | tableName from config file or metadata table and write the data to destination. Thanks. On Mon, Sep 30, 2019 at 8:56 AM swetha kadiyala <swethakadiyal...@gmail.com> wrote: > dear friends, > > I am new to spark. can you please help me to read the below text file > using spark and scala. > > Sample data > > bret|lee|A|12345|ae545|gfddfg|86786786 > 142343345|aaaa|D|ae342 > 67567|67777|U|aadfsd|34k4|84304|020|sdnfsdfn|3243|benej|32432|jsfsdf|3423 > 67564|67747|U|aad434|3435|843454|203|sdn454dfn|3233|gdfg|34325|sdfsddf|7657 > > > I am receiving indicator type with 3 rd column of each row. if my > indicator type=A, then i need to store that particular row data into a > table called Table1. > if indicator type=D then i have to store data into seperate table called > TableB and same as indicator type=U then i have to store all rows data into > a separate table called Table3. > > Can anyone help me how to read row by row and split the columns and apply > the condition based on indicator type and store columns data into > respective tables. > > Thanks, > Swetha >
