devinjdangelo commented on issue #7228:
URL:
https://github.com/apache/arrow-datafusion/issues/7228#issuecomment-1670311788
I agree UX needs work. An additional problem is `ListingTableInsertMode
`cannot be controlled from SQL or sesision config parameters. So while you can
append new files to a table via rust like so:
```rust
ctx.register_json(
"json_table_sink",
out_path,
NdJsonReadOptions::default()
.insert_mode(ListingTableInsertMode::AppendNewFiles)
)
.await?;
ctx.sql("insert into json_table_sink ....")
```
Registering a table via SQL doesn't expose this setting, so it always
defaults to ListingTableInsertModeAppendToFile.
I am thinking Option3 makes the most sense to solve these issues. My concern
with option2 is we will have to add a lot of new SQL grammar to support these
and future write options (e.g. compression, parquet row group size ect...). For
parquet read configs, we have a lot of options controlled via session config,
and we could follow a similar pattern for default write behaviors (both parquet
and non parquet specific write behaviors).
For option 2, maybe some syntax like `insert into table WITH (...) `where
(...) is arbitrary config tuples could work. I believe [trino
](https://trino.io/docs/current/sql/create-table.html) has some syntax like
this for creating tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
