xuanyuanking commented on pull request #30521: URL: https://github.com/apache/spark/pull/30521#issuecomment-738054523
Em... I just reviewed my comment in https://github.com/apache/spark/pull/30521#issuecomment-737871813. I didn't mention the assumption. Actually, I almost finish the code for the default flag, but it might be better not to add more API or extra flag based on the current sink behavior. ``` The existence of SPARK-33638 means you must provide the way ``` Sorry, maybe we have a misunderstanding about the meaning of SPARK-33638. I fully agree with you on the support for the V2 table, so I created the Jira and kept adding the discussion in this PR to the Jira comment. I created SPARK-33638 to track the issue we need to solve for the V2 support, but this PR is for V1. I don't think if there's something that can't be supported in the V2 table now, then we need to block V1 support. Do you agree? ------ (Here I give an issue that needs further discussion for the V2 support. If you don't care about the detail, please ignore. The discussion was also commented in SPARK-33638) Per https://github.com/apache/spark/pull/30521#discussion_r533869095, for the partition column. Now the param only takes effects in the v1 sink. How do we address the conflict between user input and the partitioning for the existing table? If we want to let the data source decide, the V2 plan didn't carry the partition info for now. So we can mark down all the issues we already found and fix them one by one in SPARK-33638. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
