beliefer commented on issue #23950: [SPARK-27140][SQL]The feature is 'insert overwrite local directory' has an inconsistent behavior in different environment. URL: https://github.com/apache/spark/pull/23950#issuecomment-475466561 @srowen Thanks a lot! I can always see your reply, as I expected. The whole process is below： First, the path /tmp/noexistdir is not exists and the following test cases occur in local mode. Second, when Maropu was running insert overwrite local directory '/tmp/noexistdir/t' select * from t in his environment, the not exists path would be created and result files and file _SUCCESS would be moved to the path. Third, when I was running insert overwrite local directory '/tmp/noexistdir/t' select * from t in my environment, the not exists path would be created as a single file and the single file is one of the result files. Second, I created this PR to execute the same test case in UT. When UT was running insert overwrite local directory '/tmp/noexistdir/t' select * from t in jenkins environment, the behavior is the same as the first test case. Why the feature 'insert overwrite local directory' has an inconsistent behavior in different environment? According to the source code of InsertIntoHiveDirCommand, there no chance to create the target path that doesn't exist yet. Could you help me, find out the reason. Thanks!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
