andygrove commented on code in PR #2784:
URL:
https://github.com/apache/datafusion-ballista/pull/2784#discussion_r4601705674
##########
integrations/iceberg/src/catalog.rs:
##########
@@ -145,6 +159,139 @@ impl SchemaProvider for PinningSchema {
}
}
+/// A catalog-backed table: its catalog, by address, and its identifier.
+type TableKey = (usize, TableIdent);
+
+fn key(provider: &IcebergCatalogTableProvider) -> TableKey {
+ let catalog = Arc::as_ptr(provider.catalog()).cast::<()>() as usize;
+ (catalog, provider.table_ident().clone())
+}
+
+async fn load(provider: &IcebergCatalogTableProvider) -> Result<Table> {
+ provider
+ .catalog()
+ .load_table(provider.table_ident())
+ .await
+ .map_err(to_datafusion_error)
+}
+
+/// The catalog-backed table a scan's pinned table was pinned from, if any.
+fn origin(scan: &TableScan) -> Option<&IcebergCatalogTableProvider> {
Review Comment:
One case I wanted to check with you. Say `t`'s catalog is briefly
unreachable and someone runs `CREATE VIEW w AS SELECT * FROM v` meanwhile. Then
`v` resolves to a `RepinnedView` holding the error, and since it has no plan,
`w` stores a plain `TableScan` of that `RepinnedView`. Later lookups of `w`
find no `PinnedTable` here, so `repin` returns `None` and `w` keeps failing
with the old error, even once `t` loads fine again and `v` itself works. Would
it make sense for `repin` to also recognise a scan over a `RepinnedView` and
re-pin its `original`'s plan? Or is this rare enough to just note in the docs?
##########
integrations/iceberg/tests/scan.rs:
##########
@@ -42,36 +44,159 @@ mod standalone {
assert_standalone_scans(memory_catalog(dir.path()).await).await
}
- // A known limitation: a view stores its planned query, and tables are
- // pinned when a query is planned.
#[tokio::test]
- async fn should_keep_a_views_version() -> Result<()> {
+ async fn should_read_a_views_latest_version() -> Result<()> {
let dir = tempfile::tempdir()?;
- let catalog = memory_catalog(dir.path()).await;
- let state = SessionStateBuilder::new()
- .with_config(SessionConfig::new_with_ballista())
- .with_default_features()
- .build();
- let ctx =
SessionContext::standalone_with_state(register_iceberg(state)).await?;
- ctx.register_table("t", Arc::new(catalog_table(&catalog).await))?;
+ let (catalog, ctx) = view_context(dir.path()).await?;
insert(&catalog, "(1, 'a')").await;
ctx.sql("CREATE VIEW v AS SELECT * FROM t").await?;
+ // A view over a view, reading `t` again in a subquery.
+ ctx.sql("CREATE VIEW w AS SELECT * FROM v WHERE id IN (SELECT id FROM
t)")
+ .await?;
insert(&catalog, "(2, 'b')").await;
- let result = ctx.sql("SELECT * FROM v").await?.collect().await?;
+ for view in ["v", "w"] {
+ let sql = format!("SELECT * FROM {view} ORDER BY id");
+ let result = ctx.sql(&sql).await?.collect().await?;
+ assert_batches_eq!(LATEST, &result);
+ }
+
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn should_read_a_dataframe_views_latest_version() -> Result<()> {
+ let dir = tempfile::tempdir()?;
+ let (catalog, ctx) = view_context(dir.path()).await?;
+
+ insert(&catalog, "(1, 'a')").await;
+ let df = ctx.sql("SELECT * FROM t").await?;
+ ctx.register_table("v", df.clone().into_view())?;
+ ctx.register_table("tv", df.into_temporary_view())?;
+ insert(&catalog, "(2, 'b')").await;
+
+ for (view, table_type) in [("v", TableType::View), ("tv",
TableType::Temporary)] {
+ let sql = format!("SELECT * FROM {view} ORDER BY id");
+ let result = ctx.sql(&sql).await?.collect().await?;
+ assert_batches_eq!(LATEST, &result);
+ assert_eq!(ctx.table_provider(view).await?.table_type(),
table_type);
+ }
+
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn should_fail_to_read_a_view_whose_table_is_gone() -> Result<()> {
+ let dir = tempfile::tempdir()?;
+ let (catalog, ctx) = view_context(dir.path()).await?;
+ ctx.sql("CREATE VIEW v AS SELECT * FROM t").await?;
+ drop_table(&catalog).await;
+
+ // Without a filter, the client fails to encode the view's scan.
+ let error = ctx
+ .sql("SELECT * FROM v")
+ .await?
+ .collect()
+ .await
+ .unwrap_err();
+ assert_eq!(
+ without_location(&error.to_string()),
Review Comment:
Small suggestion. Asserting the full string ties this test to the
composed-codec layout and to DataFusion's "likely caused by a bug" boilerplate,
so an unrelated change to either would break it. Would checking that it
contains `No such table` be enough here? Separately, that boilerplate comes
from `distributed_query.rs` wrapping every encode error as `Internal`, which
isn't a great message for a missing table. I think that's worth its own small
issue rather than something for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]