Is there a way how to change the default storage level?
If not, how can I properly change the storage level wherever necessary, if
my input and intermediate results do not fit into memory?
In this example:
context.wholeTextFiles(...)
.flatMap(s -> ...)
.flatMap(s -> ...)
Does persist() need to be called after every transformation?
context.wholeTextFiles(...)
.persist(StorageLevel.MEMORY_AND_DISK)
.flatMap(s -> ...)
.persist(StorageLevel.MEMORY_AND_DISK)
.flatMap(s -> ...)
.persist(StorageLevel.MEMORY_AND_DISK)
Thanks!