Hi, Any advice how to do this in spark sql ? I have a scenario as below
dataframe1 = loaded from an HDFS Parquet file. dataframe2 = read from a Kafka Stream. If column1 of dataframe1 value in columnX value of dataframe2 , then I need then I need to replace column1 value of dataframe1. Else add column1 value of dataframe1 to dataframe2 as a new record. In a sense need to implement a look up dataframe which is refresh-able. For more information please check https://stackoverflow.com/questions/57479581/how-to-do-this-scenario-in-spark-streaming?noredirect=1#comment101437596_57479581 Let me know if u need more info Thanks
