NoahKusaba opened a new issue, #2882:
URL: https://github.com/apache/datafusion-ballista/issues/2882
**Describe the bug**
With `ballista-iceberg` (#2638), a catalog-backed Iceberg table registered
in an ordinary schema with `register_table` can't be dropped with SQL once its
catalog can no longer load it, for example after the table is dropped from the
catalog or while the catalog is unreachable.
`PinningSchema::table` loads an `IcebergCatalogTableProvider` on every
lookup and returns the error if the load fails. DataFusion's `DROP TABLE` looks
the table up first (`SessionContext::find_and_deregister`), so the error makes
it report the table as missing:
- `DROP TABLE t` fails with `Table 't' doesn't exist.`
- `DROP TABLE IF EXISTS t` succeeds without dropping it, and `t` stays
registered.
**To Reproduce**
```rust
let ctx =
SessionContext::standalone_with_state(register_iceberg(state)).await?;
let t = IcebergCatalogTableProvider::try_new(catalog.clone(), namespace,
"t").await?;
ctx.register_table("t", Arc::new(t))?;
catalog.drop_table(&TableIdent::from_strs(["ns", "t"])?).await?;
ctx.sql("DROP TABLE t").await; // Err: Table 't' doesn't exist.
ctx.sql("DROP TABLE IF EXISTS t").await; // Ok, but t is still registered
```
**Expected behavior**
`DROP TABLE` removes the table, as it does in single-process DataFusion,
where `IcebergCatalogTableProvider` only loads its table in `scan()`. Reading
the table should still fail with the load error.
`ctx.deregister_table("t")` works, since it doesn't look the table up. #2784
documents that as the workaround.
**Additional context**
#2784 fixes the same problem for views: a view whose tables can't be loaded
is returned as a `RepinnedView` that holds the error, so it can be dropped or
replaced and fails only when read. Fix options:
- A similar wrapper for base tables that reports the original's type and
schema and fails on scan and encode. `PinnedTable` is what the codec and the
view re-pinning look for, so they would have to see through the wrapper.
- Pin when a query runs instead of at lookup (#2881), after which lookups do
no I/O and this goes away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]