Joe McDonnell created IMPALA-6899:
-------------------------------------
Summary: Dataload uses excessive HDFS commands
Key: IMPALA-6899
URL: https://issues.apache.org/jira/browse/IMPALA-6899
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Joe McDonnell
Assignee: Joe McDonnell
Dataload has several locations where it does a long string of HDFS commands
similar to this:
{code:java}
hdfs dfs -mkdir bad_table1
hdfs dfs -put bad_file_1 bad_table1
hdfs dfs -put bad_file_2 bad_table1
hdfs dfs -mkdir bad_table2
hdfs dfs -put bad_file_3 bad_table2
hdfs dfs -put bad_file_4 bad_table2{code}
Most hdfs shell commands can take multiple arguments. In particular, "mkdir"
can make multiple directories in one command. "put" can copy multiple files
into a single destination. This can save on hdfs commandline invocations, which
are often expensive due to JVM startup and other costs. For example, the above
is equivalent to:
{code:java}
hdfs dfs -mkdir bad_table1 bad_table2
hdfs dfs -put bad_file_1 bad_file_2 bad_table1
hdfs dfs -put bad_file_3 bad_file_4 bad_table2{code}
Dataload should make these types of optimizations wherever possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
