Dear Sir or Madam:
I am a Livy beginner. I use Livy, because within an interactive session,
different spark jobs could share cached RDDs or DataFrames.
When I read some parquet files and create a table called “TmpTable”. The
following queries will use this table. Does it mean this table has been cached?
If cached, where is the table cached? The table is cached in Livy or
Spark also supports cache function. When I read some parquet files and
create a table called “TmpTable2”. I add such code:
In the next query using this table. It will be cached in Spark cluster.
Then the following queries could use this cached table.
What is the difference between cached in Livy and cached in Spark cluster?