Sailesh Mukil has uploaded a new change for review. http://gerrit.cloudera.org:8080/5449
Change subject: IMPALA-4611: Checking perms on S3 files is a very expensive no-op ...................................................................... IMPALA-4611: Checking perms on S3 files is a very expensive no-op We call getPermissions() on partition directories to find out if Impala has access to those files. On S3, this currently is a no-op as the S3A connector does not try to set/get the permissions for S3 objects. So, it always returns the default set of permissions -> 777. However, it still makes a roundtrip to S3 causing a slow down in the Catalog. We can return the READ_WRITE permission immediately if we know we are accessing an S3 file, thereby avoiding the round trip to S3 for every partition. This will greatly speedup metadata operations for S3 tables and partitions, which is already known to be a big bottleneck. If and when the S3A connector is able to manage permissions in the future, we need to revisit this code. However, as permissions on S3 are unsupported by Impala right now, we might as well gain on perf. Change-Id: If9d1072c185a6162727019cdf1cb34d7f3f1c75c --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 1 file changed, 9 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/5449/1 -- To view, visit http://gerrit.cloudera.org:8080/5449 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If9d1072c185a6162727019cdf1cb34d7f3f1c75c Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Sailesh Mukil <[email protected]>
