Hi, what is the version of spark are you using? And where is the data stored.
I am not quite sure that just using a bash script will help because concatenating all the files into a single file creates a valid JSON. Regards, Gourav On Tue, Apr 26, 2022 at 3:44 PM Sid <flinkbyhe...@gmail.com> wrote: > Hello, > > Can somebody help me with the below problem? > > > https://stackoverflow.com/questions/72015557/dealing-with-large-number-of-small-json-files-using-pyspark > > > Thanks, > Sid >
