Sergio Peña created HIVE-14271:
----------------------------------
Summary: FileSinkOperator should not rename files to final paths
when S3 is the default destination
Key: HIVE-14271
URL: https://issues.apache.org/jira/browse/HIVE-14271
Project: Hive
Issue Type: Sub-task
Reporter: Sergio Peña
Assignee: Sergio Peña
FileSinkOperator does a rename of {{outPaths -> finalPaths}} when it finished
writing all rows to a temporary path. The problem is that S3 does not support
renaming.
Two options can be considered:
a. Use a copy operation instead. After FileSinkOperator writes all rows to
outPaths, then the commit method will do a copy() call instead of move().
b. Write row by row directly to the S3 path (see HIVE-1620). This may add
better performance calls, but we should take care of the cleanup part in case
of writing errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
