NoahKusaba opened a new issue, #2882:
URL: https://github.com/apache/datafusion-ballista/issues/2882

   **Describe the bug**
   
   With `ballista-iceberg` (#2638), a catalog-backed Iceberg table registered 
in an ordinary schema with `register_table` can't be dropped with SQL once its 
catalog can no longer load it, for example after the table is dropped from the 
catalog or while the catalog is unreachable.
   
   `PinningSchema::table` loads an `IcebergCatalogTableProvider` on every 
lookup and returns the error if the load fails. DataFusion's `DROP TABLE` looks 
the table up first (`SessionContext::find_and_deregister`), so the error makes 
it report the table as missing:
   
   - `DROP TABLE t` fails with `Table 't' doesn't exist.`
   - `DROP TABLE IF EXISTS t` succeeds without dropping it, and `t` stays 
registered.
   
   **To Reproduce**
   
   ```rust
   let ctx = 
SessionContext::standalone_with_state(register_iceberg(state)).await?;
   let t = IcebergCatalogTableProvider::try_new(catalog.clone(), namespace, 
"t").await?;
   ctx.register_table("t", Arc::new(t))?;
   catalog.drop_table(&TableIdent::from_strs(["ns", "t"])?).await?;
   
   ctx.sql("DROP TABLE t").await;           // Err: Table 't' doesn't exist.
   ctx.sql("DROP TABLE IF EXISTS t").await; // Ok, but t is still registered
   ```
   
   **Expected behavior**
   
   `DROP TABLE` removes the table, as it does in single-process DataFusion, 
where `IcebergCatalogTableProvider` only loads its table in `scan()`. Reading 
the table should still fail with the load error.
   
   `ctx.deregister_table("t")` works, since it doesn't look the table up. #2784 
documents that as the workaround.
   
   **Additional context**
   
   #2784 fixes the same problem for views: a view whose tables can't be loaded 
is returned as a `RepinnedView` that holds the error, so it can be dropped or 
replaced and fails only when read. Fix options:
   
   - A similar wrapper for base tables that reports the original's type and 
schema and fails on scan and encode. `PinnedTable` is what the codec and the 
view re-pinning look for, so they would have to see through the wrapper.
   - Pin when a query runs instead of at lookup (#2881), after which lookups do 
no I/O and this goes away.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to