Sailesh Mukil has posted comments on this change. Change subject: IMPALA-3452: S3: Disable Impala staging for INSERTs via flag for speedup ......................................................................
Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/2905/4/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: Line 865: } else if (!is_s3_path || !query_ctx_.request.query_options.s3_skip_insert_staging) { Added this to the patch. This skips making unnecessary calls to S3 to check if the path exists before creating it. The "directories" are already created by the table sinks. This gave a speedup of ~3.5 minutes on a query that creates 2.4K partitions. (Query time came down from 27 mins to 23.5 mins). Still really slow because of the catalog stat-ing every file. -- To view, visit http://gerrit.cloudera.org:8080/2905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iff9620d41ba0d5fb1aa0c9f4abb48866fc2b0698 Gerrit-PatchSet: 4 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Marcel Kornacker <[email protected]> Gerrit-Reviewer: Mostafa Mokhtar <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-HasComments: Yes
