Hello, I am new to Spark and have only used Hadoop in the past.
I understand Spark is in memory as compare to Hadoop who use disk for intermediate storage. From the practical term, the benefit must be performance, but what would be the drawbacks? e.g. - node failure? - not able to handle if intermediate data > memory size of a node - cost I would like to hear your experience when using Spark to handle big data, and what is the work around in the above cases? Thanks.
