[
https://issues.apache.org/jira/browse/HDDS-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang updated HDDS-15765:
-----------------------------------
Description:
On an HA OM cluster, {{ozone repair om compact --column-family <CF>}} and
{{ozone admin om snapshot defrag}} without {{--node-id}} do not fail cleanly,
they hang for a long time before eventually timing out.
Root cause: with no {{--node-id}}/{{--service-id}},
{{OMNodeDetails.getOMNodeDetailsFromConf(conf, null, null)}} falls back to the
base config key {{ozone.om.address}}, which in an HA deployment defaults to
{{0.0.0.0:9862}}. That default is the OM *server bind* address (bind to all
interfaces), and is meaningless as a *client* target. Because it resolves to a
non-null (but useless) address, the tool builds a valid-looking
{{OMNodeDetails}} for {{0.0.0.0}} and proceeds to the RPC. The connection to
{{0.0.0.0}} fails and retries in the Hadoop IPC retry loop, so the command
blocks. A thread dump shows {{main}} stuck in
{{Client$Connection.handleConnectionFailure}} under
{{OMAdminProtocolClientSideImpl.compactOMDB}}.
Related to HDDS-15733 (fixed the "om node: null" display and added a null-guard
for the single-OM case). That fix does not cover this HA case, because here the
address resolves to {{0.0.0.0}} rather than null, so the guard is not
triggered. {{DefragSubCommand}} shares the same latent bug (it only guards
{{omNodeDetails == null}}).
Fix: in both CLI commands, when {{--node-id}} is omitted but the config defines
an HA service ({{OmUtils.isServiceIdsDefined(conf)}}), fail fast asking the
operator to specify {{--node-id}}, instead of resolving to {{0.0.0.0}}. The
guard is kept in the CLI layer (not in the shared {{getOMNodeDetailsFromConf}}
helper) so that the message can name the {{--node-id}} flag and the legitimate
non-HA single-OM {{(null, null)}} resolution to the operator-set
{{ozone.om.address}} is preserved.
Repro: {{compose/ozone-ha}}, run either command in an {{om}} container without
{{--node-id}}; observe the hang.
was:
On an HA OM cluster, {{ozone repair om compact --column-family <CF>}} without
{{--node-id}} does not fail cleanly, it hangs for a long time before eventually
timing out.
Root cause: with no {{--node-id}}/{{--service-id}},
{{OMNodeDetails.getOMNodeDetailsFromConf(conf, null, null)}} falls back to the
base config key {{ozone.om.address}}, which in an HA deployment defaults to
{{0.0.0.0:9862}}. Because that resolves to a non-null (but useless) address,
the tool builds a valid-looking {{OMNodeDetails}} for {{0.0.0.0}} and proceeds
to the {{compactDB}} RPC. The connection to {{0.0.0.0}} fails and retries in
the Hadoop IPC retry loop, so the command blocks. A thread dump shows {{main}}
stuck in {{Client$Connection.handleConnectionFailure}} under
{{OMAdminProtocolClientSideImpl.compactOMDB}}.
Related to HDDS-15733 (fixed the "om node: null" display and added a null-guard
for the single-OM case). That fix does not cover this HA case, because here the
address resolves to {{0.0.0.0}} rather than null, so the guard is not triggered.
Suggested fix: when {{--node-id}} is omitted but the config defines multiple OM
nodes (key {{ozone.om.nodes.[serviceId]}}), fail fast asking the operator to
specify {{--node-id}}, instead of defaulting to {{0.0.0.0}}.
Repro: {{compose/ozone-ha}}, run the command in an {{om}} container without
{{--node-id}}; observe the hang.
Summary: CompactOMDB/DefragSubCommand hang when --node-id is omitted on
an HA OM cluster (was: CompactOMDB hangs when --node-id is omitted on an HA OM
cluster)
> CompactOMDB/DefragSubCommand hang when --node-id is omitted on an HA OM
> cluster
> -------------------------------------------------------------------------------
>
> Key: HDDS-15765
> URL: https://issues.apache.org/jira/browse/HDDS-15765
> Project: Apache Ozone
> Issue Type: Bug
> Components: Tools
> Reporter: Chi-Hsuan Huang
> Assignee: Chi-Hsuan Huang
> Priority: Minor
>
> On an HA OM cluster, {{ozone repair om compact --column-family <CF>}} and
> {{ozone admin om snapshot defrag}} without {{--node-id}} do not fail cleanly,
> they hang for a long time before eventually timing out.
> Root cause: with no {{--node-id}}/{{--service-id}},
> {{OMNodeDetails.getOMNodeDetailsFromConf(conf, null, null)}} falls back to
> the base config key {{ozone.om.address}}, which in an HA deployment defaults
> to {{0.0.0.0:9862}}. That default is the OM *server bind* address (bind to
> all interfaces), and is meaningless as a *client* target. Because it resolves
> to a non-null (but useless) address, the tool builds a valid-looking
> {{OMNodeDetails}} for {{0.0.0.0}} and proceeds to the RPC. The connection to
> {{0.0.0.0}} fails and retries in the Hadoop IPC retry loop, so the command
> blocks. A thread dump shows {{main}} stuck in
> {{Client$Connection.handleConnectionFailure}} under
> {{OMAdminProtocolClientSideImpl.compactOMDB}}.
> Related to HDDS-15733 (fixed the "om node: null" display and added a
> null-guard for the single-OM case). That fix does not cover this HA case,
> because here the address resolves to {{0.0.0.0}} rather than null, so the
> guard is not triggered. {{DefragSubCommand}} shares the same latent bug (it
> only guards {{omNodeDetails == null}}).
> Fix: in both CLI commands, when {{--node-id}} is omitted but the config
> defines an HA service ({{OmUtils.isServiceIdsDefined(conf)}}), fail fast
> asking the operator to specify {{--node-id}}, instead of resolving to
> {{0.0.0.0}}. The guard is kept in the CLI layer (not in the shared
> {{getOMNodeDetailsFromConf}} helper) so that the message can name the
> {{--node-id}} flag and the legitimate non-HA single-OM {{(null, null)}}
> resolution to the operator-set {{ozone.om.address}} is preserved.
> Repro: {{compose/ozone-ha}}, run either command in an {{om}} container
> without {{--node-id}}; observe the hang.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
