All transactions in JSON, It is not a single array. On Thu, Jun 18, 2020 at 12:55 PM Stephan Wehner <step...@buckmaster.ca> wrote:
> It's an interesting problem. What is the structure of the file? One big > array? On hash with many key-value pairs? > > Stephan > > On Thu, Jun 18, 2020 at 6:12 AM Chetan Khatri <chetan.opensou...@gmail.com> > wrote: > >> Hi Spark Users, >> >> I have a 50GB of JSON file, I would like to read and persist at HDFS so >> it can be taken into next transformation. I am trying to read as >> spark.read.json(path) but this is giving Out of memory error on driver. >> Obviously, I can't afford having 50 GB on driver memory. In general, what >> is the best practice to read large JSON file like 50 GB? >> >> Thanks >> > > > -- > Stephan Wehner, Ph.D. > The Buckmaster Institute, Inc. > 2150 Adanac Street > Vancouver BC V5L 2E7 > Canada > Cell (604) 767-7415 > Fax (888) 808-4655 > > Sign up for our free email course > http://buckmaster.ca/small_business_website_mistakes.html > > http://www.buckmaster.ca > http://answer4img.com > http://loggingit.com > http://clocklist.com > http://stephansmap.org > http://benchology.com > http://www.trafficlife.com > http://stephan.sugarmotor.org (Personal Blog) > @stephanwehner (Personal Account) > VA7WSK (Personal call sign) >
