[
https://issues.apache.org/jira/browse/IMPALA-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998666#comment-16998666
]
Sahil Takiar commented on IMPALA-9224:
--------------------------------------
Discussed this a bit offline with Thomas.
I think the current design of node blacklisting has an assumption that the
statestore will eventually evict bad nodes from the cluster. However, if we add
faulty disk detection to node blacklisting, that assumption probably no longer
holds. IMPALA-7872 might help here, but having extended health checks that
monitor the health of all disks on a node might not be the best solution. Disk
failures might manifest themselves in odd ways, and "disk health checks" might
not be able to capture all disk issues. For example, Spark's blacklisting
feature calls out the following disk failure scenario
(https://blog.cloudera.com/blacklisting-in-apache-spark/):
{quote}
One disk on one host has an intermittent failure that causes some sectors to
not be read. The disk is still “available” to the operating system, and other
reads and writes succeed, but reads and writes related to execution tend to
fail. This leads to a task failure.
{quote}
Assuming we decide to not add disk health checks (IMPALA-7872), there are a few
things we might want to re-consider for node blacklisting:
* Can a node be blacklisted indefinitely? If say, a node is blacklisted 10
consecutive times, is there a point in un-blacklisting it, or should it just
remain blacklisted
* Should blacklisting be able to remove a node from a cluster?
> Blacklist nodes with faulty disks
> ---------------------------------
>
> Key: IMPALA-9224
> URL: https://issues.apache.org/jira/browse/IMPALA-9224
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Sahil Takiar
> Priority: Critical
>
> Similar to IMPALA-8339 and IMPALA-9137, Impala should blacklist nodes with
> faulty disks. Specifically, if a query fails because of a disk error, the
> node with that disk should be blacklisted and the query should be retried.
> We shouldn't need to blacklist nodes that fail to read from HDFS / S3, since
> they contain their own internal mechanisms for recovering from faulty disks.
> We should only blacklist nodes when failing to read / write from *local*
> disks.
> The two main components of Impala that read / write from local disk are the
> spill-to-disk and data caching features. Whenever a query fails because of a
> disk failure during spill-to-disk, the node should be blacklisted.
> Reads / writes from / to the data cache are a bit different. If a cache read
> fails due to a disk error, the error will be printed out and the Lookup()
> call to the cache will return 0 bytes read, which means it couldn't find the
> data in the cache. This should cause the scan to fall back to a normal,
> un-cached read. While this doesn't affect query correctness or the ability
> for a query to complete, it can affect performance. Since cache failures
> don't result in query failures, we might consider having a threshold of data
> cache read / writes errors before blacklisting a node.
> We need to be careful to only capture specific disk failures - e.g. disk
> quota, permission denied, etc. errors shouldn't result in blacklisting as
> they typically are a result of system misconfiguration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
