cloud-fan commented on pull request #30521: URL: https://github.com/apache/spark/pull/30521#issuecomment-737038157
`DataFrameWriterV2` is very powerful to describe the table writing behavior (CREATE, CREATE IF NOT EXISTS, CREATE OR REPLACE, REPLACE, append, overwrite where, etc.) and I don't think the current streaming framework can support these at the current stage. Ideally we need to handle these cases: 1. table exists and users want to write to it 2. table not exists and users want to fail 3. table not exists and users want to create it The current PR can't cover case 2 but I don't know how common it is for streaming users. Adding a `DataStreamWriterV2` to cover case 2 looks an overkill to me. One possible solution is to add 2 methods `insertTable` and `createAndInsertTable`. If we think case 2 is rare, adding only `toTable` which works as `createAndInsertTable` is also fine to me. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
