alamb commented on issue #5130: URL: https://github.com/apache/arrow-datafusion/issues/5130#issuecomment-1430323461
> We will start working on this feature quite soon, so any help would be welcomed. Our team can provide initial design as well. I am certainly interested in this feature and will standby to review designs and code I terms of the API to support I would like to recommend we follow some other implementation's API than inventing our own. For example, perhaps we can use the `COPY orders TO 'orders' (FORMAT PARQUET, PARTITION_BY (year, month));` style command described by DuckDB: https://duckdb.org/2023/02/13/announcing-duckdb-070.html Using `create external table` seems like it may make implementations more complicated for some reasons: 1. It requires two commands to write files rather than just one 1. What happens if you try to write data to a table that already has a backing file 2. What happens if you issue two `INSERT INTO ` commands in a row For a format like CSV or NDJSON appending new data might be straightforward, but parquet doesn't really support append well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
