zsxwing edited a comment on pull request #30521: URL: https://github.com/apache/spark/pull/30521#issuecomment-738542179
> I'm not sure why you're still considering these users as "second class". I didn't say this. I meant these users need to create the table before starting the query no matter which behavior we decide. > I tend to concern about the "surprise" moment (if we can imagine in prior). I concern more about the possibility of table being created mistakenly without proper options. Could you give an example? For people familiar with DataFrameWriterV2, when they try to use APIs (such as `partitionedBy` and `tableProperty`) in DataStreamWriter, they will quickly notice that DataStreamWriter doesn't have such APIs, and notice the limitations of `toTable`. > Even I stepped back about default behavior if we really want to retain only one method, but still enable end users to claim avoiding creating table. Is it too hard for us to do that? If you meant adding a new method `def toTable(tableName: String, ifNotExist: Boolean): StreamingQuery`, then it might affect our future work. For example, we would need to explain how `ifNotExist` works if we add options to specify how to create the table, and might need to deprecate it in future. > For sure, the lack of functionality must be documented in javadoc. Totally agree that we should document the limitations. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
