Joe McDonnell created IMPALA-6638:
-------------------------------------
Summary: File handle cache shows contention when cold
Key: IMPALA-6638
URL: https://issues.apache.org/jira/browse/IMPALA-6638
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Joe McDonnell
Assignee: Joe McDonnell
Performance tests show that when the file handle cache is cold, there is
contention on the file handle cache partition lock. This added contention is
particularly severe when there are multiple IO threads accessing the same file
(e.g. when there is a query accessing multiple Parquet columns). This is
because the IO threads all map to the same partition because they are accessing
the same file.
The contention is due to the fact that FileHandleCache::GetFileHandle() holds
the lock while it opens the file handle. This lengthens the critical section
considerably, because opening a file handle involves network traffic to the
NameNode. This contention does not exist when the cache is hot.
FileHandleCache::GetFileHandle() should drop the lock while it is opening the
file handle.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
