a potential reason might be that you are getting a classnotfound exception when you run on the cluster (due to a missing jar in your uber jar) and you are possibly silently eating up exceptions in your code. 1- you can check if there are any failed tasks 2- you can check if there are any failed executors
Without further details, this is all I can guess :) Good luck On Tue, Jul 21, 2020 at 11:39 AM Pasha Finkelshteyn < pavel.finkelsht...@gmail.com> wrote: > Hi Rachana, > > Couls you please provide us with mre details: > > Minimal repro > Spark version > Java version > Scala version > > > On 20/07/21 08:27AM, Rachana Srivastava wrote: > > I am unable to identify the root cause of why my code is missing data > when I run as spark-submit but the code works fine when I run as java > main.... Any idea > > -- > Regards, > Pasha > > Big Data Tools @ JetBrains > -- "Talkers aren’t good doers. Rest assured that we’re going there to use our hands, not our tongues." W. Shakespeare
