felixcheung commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data URL: https://github.com/apache/spark/pull/25979#issuecomment-538782181 It looks to me we should clean the directory is what the hive community is saying. ________________________________ From: Liang-Chi Hsieh <[email protected]> Sent: Thursday, October 3, 2019 1:36:59 PM To: apache/spark <[email protected]> Cc: Felix Cheung <[email protected]>; Mention <[email protected]> Subject: Re: [apache/spark] [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data (#25979) I think the semantics of INSERT OVERWRITE should be overwriting existing data with new data. By no means it leaves old data intact with new data. I think there are few options in this case: 1. delete old data like this PR currently did 2. no delete old data, throw an exception to user, saying there are old data. (I think this might be confusing to user. I am also wondering how user can react to this, if they do not access filesystem) Any ideas? @cloud-fan<https://github.com/cloud-fan> @felixcheung<https://github.com/felixcheung> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://github.com/apache/spark/pull/25979?email_source=notifications&email_token=ACENZ6ZBUB7354FYCXP5YVDQMZJWXA5CNFSM4I35N7KKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAJQIZA#issuecomment-538117220>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACENZ6YVFGNNKVGFUXH5URTQMZJWXANCNFSM4I35N7KA>.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
