Hey I am using the Json4s-Jackson parser coming with spark and parsing roughly 80m records with totally size 900mb.
But the speed is slow. It took my 50 nodes(16cores cpu,100gb mem) roughly 30mins to parse Json to use spark sql. Jackson has the benchmark saying parsing should be ms level. Any way to increase speed? I am using spark 1.4 on Hadoop 2.7 with Java 8. Thanks a lot ! --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
