Joe McDonnell created IMPALA-8581:
-------------------------------------
Summary: Switch hdfs commandline calls in dataload and tests to
use "-d" on s3 where applicable
Key: IMPALA-8581
URL: https://issues.apache.org/jira/browse/IMPALA-8581
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.3.0
Reporter: Joe McDonnell
When doing a put or copyFromLocal via the HDFS commandline, it puts the file
into a temporary file and then moves it into place. The "-d" option skips the
temporary file and copies the file directly into place. On S3, this avoids an
extra copy and can also avoid consistency issues. We should look into using the
"-d" option across our tests. Our usages of the HDFS commandline generally
don't require atomicity of this file copy, so using "-d" is unlikely to impact
HDFS tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
