Joe McDonnell resolved IMPALA-6638.
       Resolution: Fixed
    Fix Version/s: Impala 2.12.0

> File handle cache shows contention when cold
> --------------------------------------------
>                 Key: IMPALA-6638
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6638
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.12.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Major
>             Fix For: Impala 2.12.0
> Performance tests show that when the file handle cache is cold, there is 
> contention on the file handle cache partition lock. This added contention is 
> particularly severe when there are multiple IO threads accessing the same 
> file (e.g. when there is a query accessing multiple Parquet columns). This is 
> because the IO threads all map to the same partition because they are 
> accessing the same file.
> The contention is due to the fact that FileHandleCache::GetFileHandle() holds 
> the lock while it opens the file handle. This lengthens the critical section 
> considerably, because opening a file handle involves network traffic to the 
> NameNode. This contention does not exist when the cache is hot.
> FileHandleCache::GetFileHandle() should drop the lock while it is opening the 
> file handle.

This message was sent by Atlassian JIRA

Reply via email to