[
https://issues.apache.org/jira/browse/HDDS-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang updated HDDS-15765:
-----------------------------------
Description:
On an HA OM cluster, {{ozone repair om compact --column-family <CF>}} without
{{--node-id}} does not fail cleanly, it hangs for a long time before eventually
timing out.
Root cause: with no {{--node-id}}/{{--service-id}},
{{OMNodeDetails.getOMNodeDetailsFromConf(conf, null, null)}} falls back to the
base config key {{ozone.om.address}}, which in an HA deployment defaults to
{{0.0.0.0:9862}}. Because that resolves to a non-null (but useless) address,
the tool builds a valid-looking {{OMNodeDetails}} for {{0.0.0.0}} and proceeds
to the {{compactDB}} RPC. The connection to {{0.0.0.0}} fails and retries in
the Hadoop IPC retry loop, so the command blocks. A thread dump shows {{main}}
stuck in {{Client$Connection.handleConnectionFailure}} under
{{OMAdminProtocolClientSideImpl.compactOMDB}}.
Related to HDDS-15733 (fixed the "om node: null" display and added a null-guard
for the single-OM case). That fix does not cover this HA case, because here the
address resolves to {{0.0.0.0}} rather than null, so the guard is not triggered.
Suggested fix: when {{--node-id}} is omitted but the config defines multiple OM
nodes (key {{ozone.om.nodes.[serviceId]}}), fail fast asking the operator to
specify {{--node-id}}, instead of defaulting to {{0.0.0.0}}.
Repro: {{compose/ozone-ha}}, run the command in an {{om}} container without
{{--node-id}}; observe the hang.
was:
On an HA OM cluster, {{ozone repair om compact \-\-column\-family }} without
{{\-\-node\-id}} does not fail cleanly, it hangs for a long time before
eventually timing out.
Root cause: with no {{\-\-node\-id}}/{{\-\-service\-id}},
{{OMNodeDetails.getOMNodeDetailsFromConf\(conf, null, null\)}} falls back to
the base config key {{ozone.om.address}}, which in an HA deployment defaults to
{{0.0.0.0:9862}}. Because that resolves to a non\-null \(but useless\) address,
the tool builds a valid\-looking {{OMNodeDetails}} for {{0.0.0.0}} and proceeds
to the {{compactDB}} RPC. The connection to {{0.0.0.0}} fails and retries in
the Hadoop IPC retry loop, so the command blocks. A thread dump shows {{main}}
stuck in {{Client$Connection.handleConnectionFailure}} under
{{OMAdminProtocolClientSideImpl.compactOMDB}}.
Related to HDDS\-15733 \(fixed the "om node: null" display and added a
null\-guard for the single\-OM case\). That fix does not cover this HA case,
because here the address resolves to {{0.0.0.0}} rather than null, so the guard
is not triggered.
Suggested fix: when {{\-\-node\-id}} is omitted but the config defines multiple
OM nodes \({{ozone.om.nodes.}}\), fail fast asking the operator to specify
{{\-\-node\-id}}, instead of defaulting to {{0.0.0.0}}.
Repro: {{compose/ozone\-ha}}, run the command in an {{om}} container without
{{\-\-node\-id}}; observe the hang.
> CompactOMDB hangs when --node-id is omitted on an HA OM cluster
> ---------------------------------------------------------------
>
> Key: HDDS-15765
> URL: https://issues.apache.org/jira/browse/HDDS-15765
> Project: Apache Ozone
> Issue Type: Bug
> Components: Tools
> Reporter: Chi-Hsuan Huang
> Assignee: Chi-Hsuan Huang
> Priority: Minor
>
> On an HA OM cluster, {{ozone repair om compact --column-family <CF>}} without
> {{--node-id}} does not fail cleanly, it hangs for a long time before
> eventually timing out.
> Root cause: with no {{--node-id}}/{{--service-id}},
> {{OMNodeDetails.getOMNodeDetailsFromConf(conf, null, null)}} falls back to
> the base config key {{ozone.om.address}}, which in an HA deployment defaults
> to {{0.0.0.0:9862}}. Because that resolves to a non-null (but useless)
> address, the tool builds a valid-looking {{OMNodeDetails}} for {{0.0.0.0}}
> and proceeds to the {{compactDB}} RPC. The connection to {{0.0.0.0}} fails
> and retries in the Hadoop IPC retry loop, so the command blocks. A thread
> dump shows {{main}} stuck in {{Client$Connection.handleConnectionFailure}}
> under {{OMAdminProtocolClientSideImpl.compactOMDB}}.
> Related to HDDS-15733 (fixed the "om node: null" display and added a
> null-guard for the single-OM case). That fix does not cover this HA case,
> because here the address resolves to {{0.0.0.0}} rather than null, so the
> guard is not triggered.
> Suggested fix: when {{--node-id}} is omitted but the config defines multiple
> OM nodes (key {{ozone.om.nodes.[serviceId]}}), fail fast asking the operator
> to specify {{--node-id}}, instead of defaulting to {{0.0.0.0}}.
> Repro: {{compose/ozone-ha}}, run the command in an {{om}} container without
> {{--node-id}}; observe the hang.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
