Dear Sir or Madam:
 
      I am a Livy beginner. I use Livy, because within an interactive session, 
different spark jobs could share cached RDDs or DataFrames.
 
      When I read some parquet files and create a table called “TmpTable”. The 
following queries will use this table. Does it mean this table has been cached?
      If cached, where is the table cached? The table is cached in Livy or 
Spark cluster?
 
      Spark also supports cache function.  When I read some parquet files and 
create a table called “TmpTable2”. I add such code: 
sql_ctx.cacheTable('tmpTable2').
      In the next query using this table. It will be cached in Spark cluster. 
Then the following queries could use this cached table.
 
      What is the difference between cached in Livy and cached in Spark cluster?
 
Thanks!
 
Yours
Wandong
 

Reply via email to