naikaayush commented on code in PR #45:
URL: https://github.com/apache/datafusion-iceberg/pull/45#discussion_r4370619490
##########
crates/datafusion/src/schema.rs:
##########
@@ -173,38 +175,28 @@ impl SchemaProvider for IcebergSchemaProvider {
let tables = self.tables.clone();
let name_clone = name.clone();
- // Use tokio's spawn_blocking to handle the async work on a blocking
thread pool
- let result = tokio::task::spawn_blocking(move || {
- // Create a new runtime handle to execute the async work
- let rt = tokio::runtime::Handle::current();
- rt.block_on(async move {
- // Verify the input table is empty - CREATE TABLE only accepts
schema definition
- ensure_table_is_empty(&table).await?;
-
- catalog
- .create_table(&namespace, table_creation)
- .await
- .map_err(to_datafusion_error)?;
-
- // Create a new table provider using the catalog reference
- let table_provider = IcebergTableProvider::try_new(
- catalog.clone(),
- namespace.clone(),
- name_clone.clone(),
- )
- .await?;
+ block_on_catalog("create-table", async move {
Review Comment:
the fundamental issue is that `register_table` is sync, and we are trying to
work around that by using `block_on`. so i did some digging on how other table
formats implement this.
### option 1
paimon-rust implements their wrapper over SessionContext called SQLContext,
and runs
[handle_create_table](https://github.com/apache/paimon-rust/blob/4cf34379c2f56527039f7e706488582d72a8e6ab/crates/integrations/datafusion/src/sql_context.rs#L529-L543)
as part of the `sql` -> `sql_with_options` flow
> and delegates everything else to the underlying [`SessionContext`].
we could follow this similarly and directly implement create_table under
```rust
pub async fn execute_logical_plan(&self, plan: LogicalPlan) ->
Result<DataFrame> {
match plan {
LogicalPlan::Ddl(ddl) => {
match ddl {
DdlStatement::CreateExternalTable(cmd) => {
// run the async create table stuff here
}
```
### option 2
this issue has been open for quite sometime
https://github.com/apache/datafusion/issues/19634
we could make `register_table` and `deregister_table` async in upstream
datafusion.
datafusion already calls `register_table` on DDL, so it is the most logical
place to implement async create table integrations with catalog providers.
[DdlStatement::CreateExternalTable](https://github.com/apache/datafusion/blob/3c2cd09da4ad39040ea5266b6bea59bb55d1d0cf/datafusion/core/src/execution/context/mod.rs#L707-L711)
what do you think about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
