Hello Everyone, Someone asked this question on JIRA and since it was a question I requested him to check stack overflow. Personally I don't have an answer to this question so in case anyone has an idea please feel free to update the issue. I have marked it resolved for the time being but thought to take your opinion. Whenever you are free, link is here - https://issues.apache.org/jira/browse/SPARK-32527
Thanks in advance for your time. Regards, Rohit Mishra