kbendick edited a comment on issue #1485: URL: https://github.com/apache/iceberg/issues/1485#issuecomment-699568949
I'd also like to point out that, according to what little documentation I could find, the cached query plan _might_ not be invalidated if its using a hadoop / file based Catalog. Specifically, the documentation implies that there's a need to call `REFRESH` to actually invalidate path based data. The [documentation on `REFRESH`](https://spark.apache.org/docs/latest/sql-ref-syntax-aux-cache-refresh.html) has this definition: ``` REFRESH is used to invalidate and refresh all the cached data (and the associated metadata) for all Datasets that contains the given data source path. Path matching is by prefix, i.e. “/” would invalidate everything that is cached. ``` and further drives the point home that a `REFRESH` call is likely needed if things like `INSERT` etc are done to what I can only see as Hadoop based catalogs (or filesystem based catalogs) by giving this example in reference to path based resources. ``` -- The Path is resolved using the datasource's File Index. CREATE TABLE test(ID INT) using parquet; INSERT INTO test SELECT 1000; CACHE TABLE test; INSERT INTO test SELECT 100; REFRESH "hdfs://path/to/table"; ``` I hope that this investigation can shed further light on the intended behavior and our assumptions, as well as what needs to be done to ensure correctness, particularly when dealing with `Tables` or essentially non-temporary `Views`. I believe what I've found heavily agrees with @rdblue's assertions and explanations about the caching of `Datasets` to be equivalent to the caching of `temporary views`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
