Re: Update DF record with delta data in spark

2017-04-02 Thread Jörn Franke
If you trust that your delta file is correct then this might be the way forward. You just have to keep in mind that sometimes you can have several delta files in parallel and you need to apply then in the correct order or otherwise a deleted row might reappear again. Things get more messy if a

Update DF record with delta data in spark

2017-04-02 Thread Selvam Raman
Hi, Table 1:(old File) name number salray Test1 1 1 Test2 2 1 Table 2: (Delta File) namenumber salray Test1 1 4 Test3 3 2 ​i do not have date stamp field in this table. Having composite key of name and number fields. Expected Result name number salray Test1 1 4