Whether cascade parameter can be exposed in unpersist

2021-06-03 Thread Tracy Le
Hi,in spark 2.4.0,unpersist a DF only un-cache the given DataSet ,and re-compile dependent cached queries after removing the cached query., just like the question in https://issues.apache.org/jira/browse/SPARK-21478. When all the Jobs are done, we unpersist the cached data, It can take a long

Help Migrating BaseRelation to Spark 3.x

2021-06-03 Thread Fitch, Simeon
Hi, I'm the tech lead on RasterFrames, which adds geospatial raster data capability to Apache Spark SQL. We are trying to migrate to Spark 3.x, and are struggling with getting our various DataSources to work, and wondered if some might share some tips on what might be going on. Most of our issues