NoahKusaba commented on code in PR #2784:
URL:
https://github.com/apache/datafusion-ballista/pull/2784#discussion_r4603807595
##########
integrations/iceberg/src/catalog.rs:
##########
@@ -145,6 +159,139 @@ impl SchemaProvider for PinningSchema {
}
}
+/// A catalog-backed table: its catalog, by address, and its identifier.
+type TableKey = (usize, TableIdent);
+
+fn key(provider: &IcebergCatalogTableProvider) -> TableKey {
+ let catalog = Arc::as_ptr(provider.catalog()).cast::<()>() as usize;
+ (catalog, provider.table_ident().clone())
+}
+
+async fn load(provider: &IcebergCatalogTableProvider) -> Result<Table> {
+ provider
+ .catalog()
+ .load_table(provider.table_ident())
+ .await
+ .map_err(to_datafusion_error)
+}
+
+/// The catalog-backed table a scan's pinned table was pinned from, if any.
+fn origin(scan: &TableScan) -> Option<&IcebergCatalogTableProvider> {
Review Comment:
Good catch, reproduced. Rather than special-casing `RepinnedView` scans in
`repin`, a table that can't be loaded is now a `PinnedTable` holding the error,
which still knows its catalog table. So a view is always inlined, `w` stores
`v`'s plan, and its next lookup reloads `t`. This also fixes #2882, since
lookups no longer fail. Covered by
`should_read_a_view_created_while_its_table_was_gone`.
##########
integrations/iceberg/tests/scan.rs:
##########
@@ -42,36 +44,159 @@ mod standalone {
assert_standalone_scans(memory_catalog(dir.path()).await).await
}
- // A known limitation: a view stores its planned query, and tables are
- // pinned when a query is planned.
#[tokio::test]
- async fn should_keep_a_views_version() -> Result<()> {
+ async fn should_read_a_views_latest_version() -> Result<()> {
let dir = tempfile::tempdir()?;
- let catalog = memory_catalog(dir.path()).await;
- let state = SessionStateBuilder::new()
- .with_config(SessionConfig::new_with_ballista())
- .with_default_features()
- .build();
- let ctx =
SessionContext::standalone_with_state(register_iceberg(state)).await?;
- ctx.register_table("t", Arc::new(catalog_table(&catalog).await))?;
+ let (catalog, ctx) = view_context(dir.path()).await?;
insert(&catalog, "(1, 'a')").await;
ctx.sql("CREATE VIEW v AS SELECT * FROM t").await?;
+ // A view over a view, reading `t` again in a subquery.
+ ctx.sql("CREATE VIEW w AS SELECT * FROM v WHERE id IN (SELECT id FROM
t)")
+ .await?;
insert(&catalog, "(2, 'b')").await;
- let result = ctx.sql("SELECT * FROM v").await?.collect().await?;
+ for view in ["v", "w"] {
+ let sql = format!("SELECT * FROM {view} ORDER BY id");
+ let result = ctx.sql(&sql).await?.collect().await?;
+ assert_batches_eq!(LATEST, &result);
+ }
+
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn should_read_a_dataframe_views_latest_version() -> Result<()> {
+ let dir = tempfile::tempdir()?;
+ let (catalog, ctx) = view_context(dir.path()).await?;
+
+ insert(&catalog, "(1, 'a')").await;
+ let df = ctx.sql("SELECT * FROM t").await?;
+ ctx.register_table("v", df.clone().into_view())?;
+ ctx.register_table("tv", df.into_temporary_view())?;
+ insert(&catalog, "(2, 'b')").await;
+
+ for (view, table_type) in [("v", TableType::View), ("tv",
TableType::Temporary)] {
+ let sql = format!("SELECT * FROM {view} ORDER BY id");
+ let result = ctx.sql(&sql).await?.collect().await?;
+ assert_batches_eq!(LATEST, &result);
+ assert_eq!(ctx.table_provider(view).await?.table_type(),
table_type);
+ }
+
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn should_fail_to_read_a_view_whose_table_is_gone() -> Result<()> {
+ let dir = tempfile::tempdir()?;
+ let (catalog, ctx) = view_context(dir.path()).await?;
+ ctx.sql("CREATE VIEW v AS SELECT * FROM t").await?;
+ drop_table(&catalog).await;
+
+ // Without a filter, the client fails to encode the view's scan.
+ let error = ctx
+ .sql("SELECT * FROM v")
+ .await?
+ .collect()
+ .await
+ .unwrap_err();
+ assert_eq!(
+ without_location(&error.to_string()),
Review Comment:
Agreed. Errors that come through plan encoding now only check for `No such
table`. Opened #2885 for the `Internal` wrapping in `distributed_query.rs`. The
composed codec also flattens the cause into its message; that's a TODO on
`ComposedLogicalExtensionCodec` and will be fixed as part of upstreaming it to
DataFusion, rather than as a separate Ballista issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]