AngersZhuuuu commented on PR #36070: URL: https://github.com/apache/spark/pull/36070#issuecomment-1091328814
Current DS insert only support write staging dir for dynamic partition overwrite, this pr's case seems is to use hive serde(since hive serde support config staging dir use different file system). And spark's commit protocol not support different file system for staging dir. Add support for different file system have to consider a lot, you can check the point mentioned in https://github.com/apache/spark/pull/33828 Also I am writing a new build in commit protocol https://github.com/apache/spark/pull/36056, it's behavior like hive, and it make all overwrite use a staging dir in the same file system of the target path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
