Joe McDonnell created IMPALA-7773:
-------------------------------------
Summary: ScanRange::ReadFromCache() holds RequestContext lock
while opening file
Key: IMPALA-7773
URL: https://issues.apache.org/jira/browse/IMPALA-7773
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Joe McDonnell
When a scanner thread is reading a file via HDFS caching, it executes
ScanRange::ReadFromCache() while holding the RequestContext::lock_.
ReadFromCache() calls Open(), which can require an RPC to the NameNode for
HDFS. If the NameNode is slow (or hangs), this will be holding the lock for the
duration. This lock is used by other scanner threads for this scan node. Disk
Io Mgr threads sometimes need to get this lock (e.g.
RequestContext::ReadDone(), called by DiskThreadLoop()). This can severely
impact the system.
We should look into what it would take to drop the lock during the Open() call
(and during the subsequent hadoopReadZero() call).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
