CHENXCHEN commented on PR #36070: URL: https://github.com/apache/spark/pull/36070#issuecomment-1091389301
> Current DS insert only support write staging dir for dynamic partition overwrite, this pr's case seems is to use hive serde(since hive serde support config staging dir use different file system). And spark's commit protocol not support different file system for staging dir. > > Add support for different file system have to consider a lot, you can check the point mentioned in #33828 > > Also I am writing a new build in commit protocol #36056, it's behavior like hive, and it make all overwrite use a staging dir in the same file system of the target path. Yes, this pr's case is to use hive serde(hive serde support config staging dir use different file system) #33828 and #36056 If we have multiple filesystems that are different from the staging dir filesystem, we still have exceptions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
