alamb commented on issue #5130: URL: https://github.com/apache/arrow-datafusion/issues/5130#issuecomment-1452082629
Here is what I heard at our meeting (documenting here in case others are following along). Please correct me if I am wrong 1. DataFusion should have some sort of Trait (separate from `TableProvider`) for "writing to a table / sink. This trait will allow other systems to implement whatever semantics they may way 2. The trait should support both "streaming" (incremental writing and appending) as well as writing complete files 3. There will be some sort of implementation in DataFusion that allows writing to the existing file formats (e.g. parquet, csv, etc) that may or may not support appending (or may support appending when there is some external coordination) but this implementation will remain simple The `INSERT INTO <...>` and `COPY ...` functionality will be implemented in terms of the trait. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
