Gargi-jais11 commented on code in PR #274: URL: https://github.com/apache/ozone-site/pull/274#discussion_r2719687712
########## docs/03-core-concepts/01-architecture/06-recon.md: ########## @@ -0,0 +1,98 @@ +--- +sidebar_label: Recon +--- + +# Recon + +Recon serves as a management and monitoring console for Ozone. It gives a bird's-eye view of Ozone and helps users troubleshoot any issues by presenting the current state of the cluster through REST based APIs and rich web UI. + +## High Level Design + + + +On a high level, Recon collects and aggregates metadata from Ozone Manager (OM), Storage Container Manager (SCM) and Datanodes (DN) and acts as a central management and monitoring console. Ozone administrators can use Recon to query the current state of the system without overloading OM or SCM. + +Recon maintains multiple databases to enable batch processing, faster querying and to persist aggregate information. It maintains a local copy of OM db and SCM db along with a SQL database for persisting aggregate information. + +Recon also integrates with Prometheus to provide a HTTP endpoint to query Prometheus for Ozone metrics and also to display a few crucial point in time metrics in the web UI. + +## Recon and Ozone Manager + + + +Recon gets a full snapshot of OM rocks db initially from the leader OM's HTTP endpoint, untars the file and initializes RocksDB for querying locally. The database is kept in sync by periodically requesting delta updates from the leader OM via RPC calls from the last applied sequence id. If for any reason, the delta updates could not be retrieved or applied to the local db, a full snapshot is requested again to keep the local db in sync with OM db. Due to this, Recon can show stale information since the local db will not always be in sync. + +The db updates retrieved from OM is then converted into a batch of events for further processing by OM db tasks via [Recon Task Framework](#task-framework). + +## Recon and Storage Container Manager + + + +Recon also acts as a passive SCM for Datanodes. When Recon is configured in the cluster, all the Datanodes register with Recon and send heartbeats, container reports, incremental container reports etc. to Recon similar to SCM. Recon uses all the information it gets from Datanodes to construct its own copy of SCM rocks db locally. Recon never sends any command to Datanodes in response and just acts as a passive SCM for faster lookup of SCM metadata. + +## Task Framework + +Recon has its own Task framework to enable batch processing of data obtained from OM and SCM. A task can listen to and act upon db events such as `PUT`, `DELETE`, `UPDATE`, etc. on either OM db or SCM db. Based on this, a task either implements `org.apache.hadoop.ozone.recon.tasks.ReconOmTask` or extends `org.apache.hadoop.ozone.recon.scm.ReconScmTask`. + +An example `ReconOmTask` is `ContainerKeyMapperTask` that persists the container -> key mapping in RocksDB. This is useful to understand which keys were part +of the container when the container is reported missing or is in a bad health state. Another example is FileSizeCountTask which keeps track of count of +files within a given file size range in a SQL database. These tasks have implementations for two scenarios: + +- Full snapshot (`reprocess()`) +- Delta updates (`process()`) + +When a full snapshot of OM db is obtained from the leader OM, the `reprocess()` is called on all the registered OM tasks. On subsequent delta updates, `process()` is called on these OM tasks. + +An example `ReconScmTask` is `ContainerHealthTask` that runs in configurable intervals to scan the list of all the containers and to persist the state of unhealthy containers (`MISSING`, `MIS_REPLICATED`, `UNDER_REPLICATED`, `OVER_REPLICATED`) in a SQL table. This information is used to determine if there are any missing containers in the cluster. + +## Recon and Prometheus + +Recon can integrate with any Prometheus instance configured to collected metrics and can display useful information in Recon UI in Datanodes and Pipelines pages. Recon also exposes a proxy endpoint +([/metrics](https://ozone.apache.org/docs/edge/interface/reconapi.html#metrics)) to query Prometheus. This integration can be enabled by setting this configuration `ozone.recon.prometheus.http.endpoint` to the Prometheus endpoint like `ozone.recon.prometheus.http.endpoint=http://prometheus:9090`. Review Comment: Do you want me to add other reconConfigKeys over here at this place or create a recon configuration page under this path http://localhost:3001/docs/administrator-guide/operations/observability/recon/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
