Vineet Garg created HIVE-21279: ---------------------------------- Summary: Avoid moving/rename operation in FileSink op for SELECT queries Key: HIVE-21279 URL: https://issues.apache.org/jira/browse/HIVE-21279 Project: Hive Issue Type: Improvement Components: Query Planning Reporter: Vineet Garg Assignee: Vineet Garg Fix For: 4.0.0 Attachments: HIVE-21279.1.patch
Currently at the end of a job FileSink operator moves/rename temp directory to another directory from which FetchTask fetches result. This is done to avoid fetching potential partial/invalid files by failed/runway tasks. This operation is expensive for cloud storage. It could be avoided if FetchTask is passed on set of files to read from instead of whole directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)