[
https://issues.apache.org/jira/browse/YUNIKORN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Krishna updated YUNIKORN-3347:
----------------------------------------
Description:
The scheduler configuration checksum is not handled robustly. Need to improve
it as follows:
* Fix checksum detection in GetConfigurationString: detect the checksum line at
the start and the end of the serialised config, and stop scanning the entire
config to locate it. The current implementation walks and rebuilds the whole
string, which is inefficient for large configs (the ones large enough to be
gzip-compressed).
* When loading a config, override an incorrect or missing checksum in the
in-memory config with the correct calculated value, and log the action taken so
a bad checksum can be traced.
* Skip processing a config update when the checksum has not changed (i.e. the
config content is effectively unchanged even if the ConfigMap was updated for
another reason). Verify this path handles a not yet set previous config.
* Make the calculated checksum reproducible from the config object: stripping
the checksum line from a serialised config must equal the config serialised
with no checksum set, so consumers can recompute the checksum without
re-running the load code.
* Expose the checksum and the checksum-comparison result in the
ValidateConfResponse returned by the /ws/v1/validate-conf REST endpoint.
Note: This changes the calculated checksum value for existing serialised
configs, causing a one time config reload on upgrade. This must be called out
in the 1.9 release notes.
was:
The scheduler configuration checksum is not handled robustly. Need to improve
it as follows:
* Fix checksum detection in GetConfigurationString: detect the checksum line at
the start and the end of the serialised config, and stop scanning the entire
config to locate it. The current implementation walks and rebuilds the whole
string, which is inefficient for large configs (the ones large enough to be
gzip-compressed).
* When loading a config, override an incorrect or missing checksum in the
in-memory config with the correct calculated value, and log the action taken so
a bad checksum can be traced.
* Skip processing a config update when the checksum has not changed (i.e. the
config content is effectively unchanged even if the ConfigMap was updated for
another reason). Verify this path handles a not-yet-set previous config.
* Make the calculated checksum reproducible from the config object: stripping
the checksum line from a serialised config must equal the config serialised
with no checksum set, so consumers can recompute the checksum without
re-running the load code.
* Expose the checksum and the checksum-comparison result in the
ValidateConfResponse returned by the /ws/v1/validate-conf REST endpoint.
Note: This changes the calculated checksum value for existing serialised
configs, causing a one time config reload on upgrade. This must be called out
in the 1.9 release notes.
> Improve scheduler config checksum generation and checks
> -------------------------------------------------------
>
> Key: YUNIKORN-3347
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3347
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler, shim - kubernetes
> Reporter: Siddharth Krishna
> Assignee: Siddharth Krishna
> Priority: Major
>
> The scheduler configuration checksum is not handled robustly. Need to improve
> it as follows:
> * Fix checksum detection in GetConfigurationString: detect the checksum line
> at the start and the end of the serialised config, and stop scanning the
> entire config to locate it. The current implementation walks and rebuilds the
> whole string, which is inefficient for large configs (the ones large enough
> to be gzip-compressed).
> * When loading a config, override an incorrect or missing checksum in the
> in-memory config with the correct calculated value, and log the action taken
> so a bad checksum can be traced.
> * Skip processing a config update when the checksum has not changed (i.e. the
> config content is effectively unchanged even if the ConfigMap was updated for
> another reason). Verify this path handles a not yet set previous config.
> * Make the calculated checksum reproducible from the config object: stripping
> the checksum line from a serialised config must equal the config serialised
> with no checksum set, so consumers can recompute the checksum without
> re-running the load code.
> * Expose the checksum and the checksum-comparison result in the
> ValidateConfResponse returned by the /ws/v1/validate-conf REST endpoint.
> Note: This changes the calculated checksum value for existing serialised
> configs, causing a one time config reload on upgrade. This must be called out
> in the 1.9 release notes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
