I have a quick newbie question. Spark Structured Streaming creates an unbounded dataframe that keeps appending rows to it.
So what's the max size of data it can hold? What if the size becomes bigger than the JVM? Will it spill to disk? I'm using S3 as storage. So will it write temp data on S3 or on local file system of the cluster? Nick
