[
https://issues.apache.org/jira/browse/ASTERIXDB-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199688#comment-16199688
]
ASF subversion and git services commented on ASTERIXDB-1076:
------------------------------------------------------------
Commit 860fcde90bd799e00eaa61cbf81badfea1c25dda in asterixdb's branch
refs/heads/master from [~mblow]
[ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=860fcde ]
[ASTERIXDB-1076][HYR] Generate heartbeats in their own thread
- Generate & send NC heartbeats in their own thread to prevent starvation
/ scheduling issues
- Fix retries on IPC connections
- Don't spin on heartbeat send failure
Change-Id: Ieae21b1596013a699f27975fb21894244c536395
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2060
Reviewed-by: Murtadha Hubail <[email protected]>
Integration-Tests: Murtadha Hubail <[email protected]>
Tested-by: Murtadha Hubail <[email protected]>
> False failures cause denying new queries
> ----------------------------------------
>
> Key: ASTERIXDB-1076
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1076
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB
> Reporter: Yingyi Bu
> Assignee: Michael Blow
> Labels: soon
>
> When CPUs in the cluster are saturated for computations, the heartbeat from
> slave nodes to the master node might get delayed. In this case, the master
> node thinks a node fails, and can no longer adds the node back. Hence, the
> entire cluster is not usable and an instance restart is needed.
> Two things need to be fixed:
> 1. (at least) expose AsterixDB configuration parameters to allow users to
> set a large heartbeat threshold;
> 2. allow a node to leave and re-join a hyracks cluster.
> In the long term, we might need to investigate better liveness check
> strategies.
> To reproduce that issue, just let slave nodes' CPUs overloaded and you will
> see that.
> The exception " Asterix Cluster Global recovery is not yet complete and The
> system is in ACTIVE state" will be thrown for upcoming queries.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
