Copilot commented on code in PR #314: URL: https://github.com/apache/ozone-site/pull/314#discussion_r2748620941
########## docs/04-user-guide/03-integrations/04-impala.md: ########## @@ -0,0 +1,83 @@ +--- +sidebar_label: Impala +--- + +# Impala + +Starting with version 4.2.0, Apache Impala provides full support for querying data stored in Apache Ozone. To utilize this functionality, ensure that your Ozone version is 1.4.0 or later. + +## Supported Access Protocols + +Impala supports the following protocols for accessing Ozone data: + +- ofs +- s3a + +Note: The o3fs protocol is **NOT** supported by Impala. + +## Supported Replication Types + +Impala is compatible with Ozone buckets configured with either: + +- **RATIS** (Replication) +- **Erasure Coding** + +## Querying Ozone Data with Impala + +Impala provides two approaches to interact with Ozone: + +- Managed Tables +- External Tables + +### Managed Tables + +If the Hive Warehouse Directory is located in Ozone, you can execute Impala queries without any changes, treating the Ozone file system like HDFS. For example: + +```sql +CREATE DATABASE d1; +``` + +```sql +CREATE TABLE t1 (x INT, s STRING); +``` + +The data will be stored under the Hive Warehouse Directory path in Ozone. + +#### Specifying a Custom Ozone Path + +You can create managed databases, tables, or partitions at a specific Ozone path using the `LOCATION` clause. Example: + +```sql +CREATE DATABASE d1 LOCATION 'ofs://ozone1/vol1/bucket1/d1.db'; +``` + +```sql +CREATE TABLE t1 LOCATION 'ofs://ozone1/vol1/bucket1/table1'; +``` + +### External Tables + +You can create an external table in Impala to query Ozone data. For example: + +```sql +CREATE EXTERNAL TABLE external_table ( + id INT, + name STRING +) +LOCATION 'ofs://ozone1/vol1/bucket1/table1'; +``` + +- With external tables, the data is expected to be created and managed by another tool. +- Impala queries the data as-is. +- The metadata is stored under the external warehouse directory. +- Note: Dropping an external table in Impala does not delete the associated data. + +## Using the S3A Protocol + +In addition to ofs, Impala can access Ozone via the S3 Gateway using the S3A file system. For more details, refer to + +- The [S3 Protocol](../01-client-interfaces/03-s3/01-s3-api.md) Review Comment: The link to the S3 Protocol documentation points to `../01-client-interfaces/03-s3/01-s3-api.md`, which does not exist in this tree (the S3 section uses `README.mdx` instead), so this will result in a broken link; update it to the correct S3 API doc path (for example, the S3 section index under `/docs/user-guide/client-interfaces/s3`). ```suggestion - The [S3 Protocol](../01-client-interfaces/03-s3/README.mdx) ``` ########## docs/03-core-concepts/03-namespace/02-buckets/06-encryption.md: ########## @@ -0,0 +1,11 @@ +--- +sidebar_label: Encryption +--- + +# Bucket Level Encryption + +## Transparent Data Encryption - TDE + +Buckets can be configured for Transparent Data Encryption (TDE) at the time of creation. +When TDE is enabled, all data written to the bucket is automatically encrypted at rest using a specified encryption key. +For detailed steps on setting up and using TDE, refer to the [Securing TDE documentation](../../../05-administrator-guide/02-configuration/03-security/05-encryption/02-transparent-data-encryption.md). Review Comment: The link to `02-transparent-data-encryption.md` under the encryption configuration directory does not correspond to a file added in this PR (only `README.mdx` and network-encryption docs are present), so this will be a broken link; either add the referenced TDE configuration page or update the link to point to the correct existing TDE documentation. ```suggestion For detailed steps on setting up and using TDE, refer to the [Securing TDE documentation](../../../05-administrator-guide/02-configuration/03-security/05-encryption/README.mdx). ``` ########## docs/03-core-concepts/02-replication/03-ratis.md: ########## @@ -0,0 +1,22 @@ +--- +sidebar_label: Ratis +--- + +# Ratis + +[Apache Ratis](https://ratis.apache.org/) is a highly customizable open-source Java implementation +of the [Raft consensus protocol](https://raft.github.io/). Raft is an easily understandable consensus +algorithm designed to manage replicated state. Unlike ZooKeeper or other Raft implementations such as etcd, +Ratis is designed as a library rather than a standalone consensus server, which simplifies its management +and integration. + +Ozone leverages Ratis to replicate system states across multiple nodes, ensuring high availability and +redundancy. When using Ratis for replication, each piece of data written by clients is replicated to 3 Datanodes. +Within Ozone, Ratis is employed in critical components such as the [Ozone Manager](../03-namespace/01-overview.md), +[Storage Container Manager](../01-architecture/01-overview.md), and Datanodes. It forms the central pillar for the +High Availability (HA) mechanisms of both the Ozone Manager (OM-HA) and Storage Container Manager (SCM-HA). + +For more detailed information, please visit the [Apache Ratis website](https://ratis.apache.org/). + +For troubleshooting Ratis problems encountered in production, please refer to +[Troubleshooting Ratis](/06-troubleshooting/13-ratis.md). Review Comment: The troubleshooting link uses the path `/06-troubleshooting/13-ratis.md`, which is inconsistent with other absolute doc links that include the `/docs` base and omit numeric prefixes; this is likely to produce a broken link in Docusaurus and should be updated to the canonical route for the Ratis troubleshooting page (for example `/docs/troubleshooting/ratis`). ```suggestion [Troubleshooting Ratis](/docs/troubleshooting/ratis). ``` ########## docs/05-administrator-guide/03-operations/09-observability/05-grafana.md: ########## @@ -0,0 +1,57 @@ +--- +sidebar_label: Grafana +--- + +# Dashboarding With Grafana + +Once Prometheus is up and running, Grana can be configured to monitor and Review Comment: The product name is misspelled as "Grana"; it should be "Grafana" for clarity and correctness. ```suggestion Once Prometheus is up and running, Grafana can be configured to monitor and ``` ########## docs/02-quick-start/01-installation/01-docker.md: ########## @@ -0,0 +1,96 @@ +--- +sidebar_label: Docker +--- + +# Try Ozone With Docker + +Apache Ozone can be quickly deployed using Docker Compose, making it ideal for development, testing, and evaluation purposes. This guide walks you through setting up a multi-node Ozone cluster using pre-built Docker images. + + + +## Prerequisites + +- [Docker Engine](https://docs.docker.com/engine/install/) - Latest stable version +- [Docker Compose](https://docs.docker.com/compose/install/) - Latest stable version + +## Running Ozone + +### Obtain the Docker Compose Configuration + +First, obtain Ozone's sample Docker Compose configuration: + +```bash +# Download the latest Docker Compose configuration file +curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml +``` + +### Start the Cluster + +Start your Ozone cluster with three Datanodes using the following command: + +```bash +docker compose up -d --scale datanode=3 +``` + +This command will: + +- Automatically pull required images from Docker Hub +- Create a multi-node cluster with the core Ozone services +- Start all components in detached mode + +### Verify the Deployment + +1. Check the status of your Ozone cluster components: + + ```bash + docker compose ps + ``` + + You should see output similar to this: + + ```bash + NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS + docker-datanode-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" datanode 14 seconds ago Up 13 seconds 0.0.0.0:32958->9864/tcp, :::32958->9864/tcp + docker-datanode-2 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" datanode 14 seconds ago Up 13 seconds 0.0.0.0:32957->9864/tcp, :::32957->9864/tcp + docker-datanode-3 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" datanode 14 seconds ago Up 12 seconds 0.0.0.0:32959->9864/tcp, :::32959->9864/tcp + docker-om-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" om 14 seconds ago Up 13 seconds 0.0.0.0:9874->9874/tcp, :::9874->9874/tcp + docker-recon-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" recon 14 seconds ago Up 13 seconds 0.0.0.0:9888->9888/tcp, :::9888->9888/tcp + docker-s3g-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" s3g 14 seconds ago Up 13 seconds 0.0.0.0:9878->9878/tcp, :::9878->9878/tcp + docker-scm-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" scm 14 seconds ago Up 13 seconds 0.0.0.0:9876->9876/tcp, :::9876->9876/tcp + ``` + +2. Check the Ozone version + + ```bash + docker compose exec om bash + ozone version + ``` + +3. Access the Ozone Recon server, which provides monitoring and management capabilities by navigating to the [Recon server home page](http://localhost:9888). + +## Configuration + +You can customize your Ozone deployment by modifying the configuration parameters in the `docker-compose.yaml` file: + +1. **Common Configurations**: Located under the `x-common-config` section +2. **Service-Specific Settings**: Found under the `environment` section of individual services + +As an example, to update the port on which Recon listens to, append the following configuration: + +```yaml +x-common-config: + ... + OZONE-SITE.XML_ozone.recon.http-address: 0.0.0.0:9090 +``` + +Refer to the [Configuring Ozone For Production](/docs/quick-start/installation/docker) page for more configuration guidelines. Review Comment: The "Configuring Ozone For Production" link currently points back to this Quick Start Docker page, which is misleading; it should link to the dedicated production configuration section (for example under `/docs/administrator-guide/configuration`) instead of being self-referential. ```suggestion Refer to the [Configuring Ozone For Production](/docs/administrator-guide/configuration) page for more configuration guidelines. ``` ########## docs/05-administrator-guide/03-operations/02-upgrade-and-downgrade.md: ########## @@ -0,0 +1,82 @@ +--- +sidebar_label: Upgrade and Downgrade +--- + +# Upgrade and Downgrade + +Ozone supports non-rolling upgrades and downgrades, where all components are stopped first, and then restarted with the upgraded or downgraded versions. + +## Upgrade States + +After upgrading components, the upgrade process is divided into two states: + +1. **Pre-finalized**: When the current components are stopped and the new versions are started, they will see that the data on disk was written by a previous version of Ozone and enter a pre-finalized state. In the pre-finalized state: + - The cluster can be downgraded at any time by stopping all components and restarting with the old versions. + - Backwards incompatible features introduced in the new version will be disallowed by the cluster. + - The cluster will remain fully operational with all functionality present in the old version still allowed. + - Any data created while pre-finalized will remain readable after downgrade. + +2. **Finalized**: When a finalize command is given to OM or SCM, they will enter a finalized state. In the finalized state: + - The cluster can no longer be downgraded. + - All new features of the cluster introduced in the new version can be used. + +### Querying finalization status + +**OM**: `ozone admin om finalizationstatus`. If using OM HA, finalization status is checked for the quorum, not individual OMs. + +**SCM**: `ozone admin scm finalizationstatus`. SCM will report that finalization is complete once it has finalized and is aware of enough finalized Datanodes to form a write pipeline. The remaining Datanodes will finalize asynchronously and be incorporated into write pipelines after informing SCM that they have finalized. + +**Datanodes**: `ozone admin datanode list` will list all Datanodes and their health state as seen by SCM. If SCM is finalized, then Datanodes whose health state is `HEALTHY` have informed SCM that they have finalized. Datanodes whose health state is `HEALTHY_READONLY` have not yet informed SCM that they have finished finalization. `HEALTHY_READONLY` (pre-finalized) Datanodes remain readable, so the cluster is operational even if some otherwise healthy Datanodes have not yet finalized. `STALE` or `DEAD` Datanodes will be told to finalize by SCM once they are reachable again. + +## Steps to upgrade and downgrade + +Starting with your current version of Ozone, complete the following steps to upgrade to a newer version of Ozone. + +1. If using OM HA and currently running Ozone 1.2.0 or greater, prepare the Ozone Manager. If OM HA is not being used, this step can be skipped. Review Comment: The placeholder argument `om-sevice-id` contains a typo; it should be `om-service-id` to match the flag name and be clearer for users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
