This is an automated email from the ASF dual-hosted git repository. ritesh pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/ozone-site.git
commit a5ee9b0614e385cc6f280544f85e47fe07d7ceec Author: Ritesh H Shukla <[email protected]> AuthorDate: Thu Mar 27 00:02:39 2025 -0700 Updates to architecture and overview page --- docs/02-quick-start/02-reading-writing-data.md | 313 +++++++++++++++++++- docs/03-core-concepts/01-architecture.md | 325 ++++++++++++++++++++- .../01-components/07-httpfs/01-overview.md | 79 +++++ .../01-components/07-httpfs/02-request-flow.md | 122 ++++++++ .../01-components/07-httpfs/README.mdx | 11 + src/css/custom.css | 50 ++++ src/pages/community/ask-a-question.md | 5 +- static/img/ozone/container.svg | 173 +++++++++++ static/img/ozone/ozone-client-interaction.svg | 209 +++++++++++++ static/img/ozone/ozone-namespace.svg | 79 +++++ static/img/ozone/ozone-storage-hierarchy-ec.svg | 142 +++++++++ .../ozone/ozone-storage-hierarchy-replicated.svg | 281 ++++++++++++++++++ static/img/ozone/recon.svg | 90 ++++++ 13 files changed, 1874 insertions(+), 5 deletions(-) diff --git a/docs/02-quick-start/02-reading-writing-data.md b/docs/02-quick-start/02-reading-writing-data.md index 39075d07..fb8c0965 100644 --- a/docs/02-quick-start/02-reading-writing-data.md +++ b/docs/02-quick-start/02-reading-writing-data.md @@ -1,3 +1,312 @@ -# Reading and Writing Data +--- +sidebar_label: Reading and Writing Data +--- -**TODO:** File a subtask under [HDDS-9856](https://issues.apache.org/jira/browse/HDDS-9856) and complete this page or section. +# Reading and Writing Data in Ozone + +Apache Ozone provides multiple interfaces for reading and writing data, catering to different use cases and client preferences. This guide explains how to use the three primary interfaces within a Docker environment: + +1. **Ozone Shell (`ozone sh`)** - The native command-line interface +2. **OFS (Ozone File System)** - Hadoop-compatible file system interface +3. **S3 API** - Amazon S3 compatible REST interface + +All examples assume you already have a running Ozone cluster using Docker Compose as described in the [Docker Installation Guide](./01-installation/01-docker.md). + +## Interface Comparison + +| Interface | Strengths | Use Cases | +| :-------------- |:---------------------------------------------------------------------------------------------| :------------------------------------------------------------------------ | +| **Ozone Shell** | - Full feature access Advanced operations Detailed metadata | - Administrative tasks Bucket/volume management Quota/ACL management | +| **OFS** | - Familiar HDFS-like commands Works with existing Hadoop applications Full cluster view | - Hadoop ecosystem integration Applications that need filesystem semantics | +| **S3 API** | - Industry standard Works with existing S3 clients Language-independent | - Web applications Multi-language environments Existing S3 applications | + +## 1. Using Ozone Shell (`ozone sh`) + +The Ozone Shell provides direct access to all Ozone features through a command-line interface. All commands follow the pattern: + +```bash +ozone sh <object-type> <action> <path> [options] +``` + +Where `<object-type>` is `volume`, `bucket`, or `key`. + +### Accessing the Ozone Shell + +To use the Ozone Shell in your Docker environment, execute commands inside the `om` or `ozone-client` container: + +```bash +# Example for Docker Compose +docker compose exec om bash +# or +docker compose exec ozone-client bash + +# Now you can run 'ozone sh' commands +``` + +### Working with Volumes + +Volumes are the top-level namespace in Ozone. + +```bash +# Create a volume +ozone sh volume create /vol1 + +# List all volumes +ozone sh volume list / + +# Get volume details +ozone sh volume info /vol1 + +# Delete a volume (must be empty) +# ozone sh volume delete /vol1 + +# Delete a volume recursively (deletes all contained buckets and keys) +# ozone sh volume delete -r /vol1 +``` + +### Working with Buckets + +Buckets are containers for keys (objects) within volumes. + +```bash +# Create a bucket +ozone sh bucket create /vol1/bucket1 + +# List all buckets in a volume +ozone sh bucket list /vol1 + +# Get bucket details +ozone sh bucket info /vol1/bucket1 + +# Delete a bucket (must be empty) +# ozone sh bucket delete /vol1/bucket1 + +# Delete a bucket recursively +# ozone sh bucket delete -r /vol1/bucket1 +``` + +### Working with Keys (Objects) + +Keys are the actual data objects stored in Ozone. + +```bash +# Create a test file locally +echo "Hello Ozone via Shell" > test_shell.txt + +# Upload a file (put source to destination) +ozone sh key put /vol1/bucket1/test_shell.txt test_shell.txt + +# Upload with specific replication type (Example: RATIS/THREE) +# ozone sh key put -t RATIS -r THREE /vol1/bucket1/key1_ratis /path/to/local/file + +# Download a file (get source to destination) +ozone sh key get /vol1/bucket1/test_shell.txt ./downloaded_shell.txt + +# Force overwrite when downloading +# ozone sh key get --force /vol1/bucket1/test_shell.txt ./downloaded_shell.txt + +# Get key information +ozone sh key info /vol1/bucket1/test_shell.txt + +# List keys in a bucket +ozone sh key list /vol1/bucket1 + +# Copy a key within Ozone (not directly supported, use put/get or other interfaces) + +# Rename a key +# ozone sh key rename /vol1/bucket1/test_shell.txt /vol1/bucket1/renamed_shell.txt + +# Delete a key +ozone sh key delete /vol1/bucket1/test_shell.txt + +# Note: In FSO buckets, deleted keys might move to trash depending on configuration. +# In OBS buckets, deletion is permanent via Ozone Shell. +``` + +## 2. Using OFS (Ozone File System) + +OFS provides a Hadoop-compatible file system interface (`ofs://`), making it seamless to use with applications designed for HDFS. + +### Accessing OFS + +You can use `ozone fs` commands (a wrapper around `hdfs dfs`) inside the `om` or `ozone-client` container: + +```bash +# Inside the OM or ozone-client container +docker compose exec om bash +# or +docker compose exec ozone-client bash +``` + +### Basic OFS Operations + +OFS uses standard Hadoop filesystem commands. + +```bash +# Create volume and bucket (using filesystem semantics) +ozone fs -mkdir -p /vol1/bucket_ofs + +# Upload a file +echo "Hello from OFS" > local_ofs.txt +ozone fs -put local_ofs.txt /vol1/bucket_ofs/ + +# Copy from local with explicit destination path +# ozone fs -copyFromLocal /path/to/local/file.txt /vol1/bucket_ofs/remote_file.txt + +# List files in a bucket +ozone fs -ls /vol1/bucket_ofs/ + +# List recursively +# ozone fs -ls -R /vol1/ + +# Download a file +ozone fs -get /vol1/bucket_ofs/local_ofs.txt ./downloaded_ofs.txt + +# Display file contents +ozone fs -cat /vol1/bucket_ofs/local_ofs.txt + +# Move a file (rename) +ozone fs -mv /vol1/bucket_ofs/local_ofs.txt /vol1/bucket_ofs/moved_ofs.txt + +# Copy a file within the filesystem +ozone fs -cp /vol1/bucket_ofs/moved_ofs.txt /vol1/bucket_ofs/copy_ofs.txt + +# Delete a file (moves to trash if enabled and bucket is FSO) +ozone fs -rm /vol1/bucket_ofs/copy_ofs.txt + +# Delete a file and skip trash +# ozone fs -rm -skipTrash /vol1/bucket_ofs/moved_ofs.txt + +# Create an empty file +ozone fs -touchz /vol1/bucket_ofs/empty_file.txt +``` + +### Advanced OFS Operations + +```bash +# Get file checksum +# ozone fs -checksum /vol1/bucket_ofs/moved_ofs.txt + +# Set replication factor for a file (use 'ozone sh key put' for setting on write) +# ozone fs -setrep -w 3 /vol1/bucket_ofs/important_file + +# Trash configuration is done in core-site.xml (see Ozone docs for details) +``` + +## 3. Using S3 API + +The S3 API provides compatibility with applications designed for Amazon S3. It's accessible via the S3 Gateway service, typically running on port `9878` in the Docker setup. + +### S3 Credentials + +In the default non-secure Docker setup, you can use any values for credentials. + +```bash +# Set environment variables (can be done outside the containers) +export AWS_ACCESS_KEY_ID=testuser +export AWS_SECRET_ACCESS_KEY=testuser-secret +export AWS_ENDPOINT_URL=http://localhost:9878 +``` +*(Note: Setting `AWS_ENDPOINT_URL` simplifies the `aws` commands below)* + +### Using AWS CLI + +The AWS CLI can be used from your local machine (if installed) or from within a container that has it. + +```bash +# Ensure AWS CLI is installed locally or use a container with it. + +# Create a bucket (maps to /s3v/<bucket-name> in Ozone namespace) +aws s3api create-bucket --bucket=s3bucket + +# List buckets +aws s3api list-buckets + +# Upload a file +echo "Hello S3" > s3_test.txt +aws s3 cp s3_test.txt s3://s3bucket/ + +# List objects in a bucket +aws s3 ls s3://s3bucket/ + +# Download a file +aws s3 cp s3://s3bucket/s3_test.txt ./downloaded_s3.txt + +# Delete an object +aws s3 rm s3://s3bucket/s3_test.txt + +# Delete a bucket (must be empty) +# aws s3api delete-bucket --bucket=s3bucket +``` + +## Cross-Interface Operations + +Ozone allows accessing the same data through different interfaces. + +### Namespace Mapping + +| Data Location | Ozone Shell Path | OFS Path | S3 Path (if linked/in s3v) | +| :------------------- | :------------------------ | :---------------------------- | :------------------------- | +| vol1/bucket1/file.txt | `/vol1/bucket1/file.txt` | `ofs://om/vol1/bucket1/file.txt` | (Needs bucket link) | +| s3v/s3bucket/file.txt | `/s3v/s3bucket/file.txt` | `ofs://om/s3v/s3bucket/file.txt` | `s3://s3bucket/file.txt` | + +*(Note: `om` in `ofs://` path refers to the Ozone Manager service address)* + +### Accessing S3 Data via Ozone Shell/OFS + +Objects created via S3 reside in the special `/s3v` volume. + +```bash +# Assuming 's3bucket' was created via S3 API and contains 's3_test.txt' + +# Access via Ozone Shell (inside om/client container) +docker compose exec om bash +ozone sh key list /s3v/s3bucket +ozone sh key get /s3v/s3bucket/s3_test.txt /tmp/from_s3.txt +exit + +# Access via OFS (inside om/client container) +docker compose exec om bash +ozone fs -ls /s3v/s3bucket/ +ozone fs -cat /s3v/s3bucket/s3_test.txt +exit +``` + +### Exposing Non-S3 Buckets via S3 (Bucket Linking) + +You can make buckets created outside `/s3v` accessible via the S3 Gateway using links. + +```bash +# Inside om/client container +docker compose exec om bash + +# Create a regular bucket +ozone sh volume create /myvol +ozone sh bucket create /myvol/mybucket + +# Create an S3-accessible link (target must exist, link name becomes S3 bucket name) +ozone sh bucket link /myvol/mybucket /s3v/linkedbucket +exit + +# Now access 'linkedbucket' via S3 CLI (outside container) +aws s3 cp local_file.txt s3://linkedbucket/ +aws s3 ls s3://linkedbucket/ +``` + +### Bucket Layouts (FSO vs OBS) + +Ozone buckets can have different internal layouts: + +1. **FILE_SYSTEM_OPTIMIZED (FSO):** Default. Better for hierarchical operations (like `ozone fs -mkdir`), supports trash for `ozone fs -rm`. Recommended for Hadoop/filesystem workloads. +2. **OBJECT_STORE (OBS):** Legacy layout. May offer slight performance benefits for flat object access patterns. No trash support. + +```bash +# Create buckets with specific layouts (inside om/client container) +# ozone sh bucket create /vol1/fso_bucket --layout FILE_SYSTEM_OPTIMIZED +# ozone sh bucket create /vol1/obs_bucket --layout OBJECT_STORE +``` +Most operations work on both, but FSO is generally preferred unless specific OBS characteristics are needed. + +## Summary + +You have learned how to perform basic read/write operations in Ozone using three different interfaces: Ozone Shell, OFS, and the S3 API. Each interface has its strengths, and Ozone's multi-protocol design allows you to choose the best tool for the job while accessing the same underlying data. diff --git a/docs/03-core-concepts/01-architecture.md b/docs/03-core-concepts/01-architecture.md index 8d88c62c..cf2351f2 100644 --- a/docs/03-core-concepts/01-architecture.md +++ b/docs/03-core-concepts/01-architecture.md @@ -1,5 +1,326 @@ # Architecture -**TODO:** File a subtask under [HDDS-9857](https://issues.apache.org/jira/browse/HDDS-9857) and complete this page or section. +Apache Ozone is a scalable, distributed, and highly available object store designed to handle billions of objects of any size. This document provides an overview of Ozone's architecture, including its core components, data organization, and operational concepts. -A high level overview of Ozone's components and what they do. +## Ozone Namespace + +Ozone organizes data in a hierarchical namespace consisting of three levels: + +### Volumes + +Volumes are the top-level entities in the Ozone namespace, conceptually similar to filesystems in traditional storage systems. They typically represent: + +- Multi-tenant boundaries +- Organizational divisions +- Project groupings + +Volumes provide isolation and can have their own admins and quota limits. For more information, see [Volumes Overview](./03-namespace/01-volumes/01-overview.md). + +### Buckets + +Buckets exist within volumes and act as containers for objects (keys). Each bucket can be configured with specific properties: + +- Storage type and replication factor +- Encryption settings +- Access control policies +- Quota limits + +Buckets are analogous to directories in a filesystem or buckets in cloud object stores. For more information, see [Buckets Overview](./03-namespace/02-buckets/01-overview.md). + +### Keys (Objects) + +Keys are the actual data objects stored in buckets. They can be: + +- Files of any size +- Binary data +- Named using a path-like structure depending on bucket layout + +For more details about Ozone's namespace, see [Namespace Overview](./03-namespace/README.mdx). + +### Ozone Bucket Types + +Ozone supports two distinct bucket layouts, each optimized for different use cases: + +#### Object Store Layout + +The Object Store layout (OBS) works like traditional object stores with a flat namespace: + +- Objects are stored with their full key path +- No concept of directories (though path-like naming is supported) +- Optimized for object storage workloads +- Compatible with S3-style access patterns + +#### File System Optimized Layout + +The File System Optimized layout (FSO) provides hierarchical directory structure: + +- Directories are first-class entities +- Supports efficient directory operations (listing, renaming) +- Includes filesystem features like trash +- Better performance for filesystem-style workloads +- Default layout type + +The bucket layout determines how data is organized and accessed within a bucket. For more information, see [Bucket Layouts](./03-namespace/02-buckets/04-layouts/README.mdx). + + + +## Core Components + +Ozone has a modular architecture with several key components that work together to provide a scalable and reliable storage system. + +### Ozone Manager (OM) + +The Ozone Manager is the metadata server that manages the namespace: + +- Handles all volume, bucket, and key operations +- Maintains metadata in RocksDB +- Allocates blocks for data storage +- Manages access control +- Supports HA deployment with Ratis consensus protocol + +The OM is the entry point for all namespace operations. It tracks which objects exist and where they are stored. For more information, see [Ozone Manager Details](../07-system-internals/01-components/01-ozone-manager/README.mdx). + +### Storage Container Manager (SCM) + +The Storage Container Manager orchestrates the container lifecycle and coordinates datanodes: + +- Manages container creation and allocation +- Tracks datanode status and health +- Handles container replication and EC +- Issues block allocation requests +- Coordinates container balancing +- Supports HA deployment with Ratis + +SCM is the control plane for container management. For more information, see [Storage Container Manager Details](../07-system-internals/01-components/02-storage-container-manager/README.mdx). + +### Datanode + +Datanodes are the workhorses that store the actual data: + +- Store data in containers on local disks +- Serve read and write requests +- Report container statistics to SCM +- Participate in replication pipelines +- Handle data integrity checks + +Each datanode manages a set of containers and serves read/write requests from clients. For more information, see [Datanode Details](../07-system-internals/01-components/03-datanode/README.mdx). + +### Recon + +Recon is the analytics and monitoring component: + +- Collects and aggregates metrics +- Provides a web UI for monitoring +- Offers a consolidated view of the cluster +- Helps identify issues and bottlenecks +- Syncs data from OM, SCM, and Datanodes + +Recon is an optional but recommended component for operational visibility. For more information, see [Recon Details](../07-system-internals/01-components/04-recon/README.mdx). + +### S3 Gateway + +The S3 Gateway provides S3-compatible API access: + +- Implements S3 REST API +- Translates S3 operations to Ozone operations +- Supports most S3 features and SDKs +- Provides authentication and authorization + +The S3 Gateway enables applications built for S3 to work with Ozone. For more information, see [S3 Gateway Details](../07-system-internals/01-components/05-s3-gateway/README.mdx). + +### HttpFS + +HttpFS provides a REST interface compatible with WebHDFS: + +- Enables HTTP access to Ozone +- Compatible with HDFS clients +- Supports read/write operations +- Facilitates integration with web applications + +HttpFS allows web applications to interact with Ozone using familiar APIs. For more information, see [HttpFS Details](../07-system-internals/01-components/07-httpfs/README.mdx). + +### Ozone Client + +The Ozone client is the software component that enables applications to interact with the Ozone storage system: + +- Provides Java libraries for programmatic access +- Handles communication with OM for namespace operations +- Manages direct data transfer with datanodes +- Implements client-side caching for improved performance +- Offers pluggable interfaces for different protocols (O3, S3, OFS) +- Handles authentication and token management + +The client library abstracts away the complexity of the distributed system, providing applications with a simple, consistent interface to Ozone storage. For more information, see [Client Details](../07-system-internals/01-components/06-client/README.mdx). + +### Component Interactions + +The components of Ozone interact in well-defined patterns for different operations: + + + +#### Write Path Sequence + +The typical write sequence follows these steps: + +1. **Namespace Operations**: The client contacts the Ozone Manager to create or locate the key in the namespace +2. **Block Allocation**: The Ozone Manager requests blocks from the Storage Container Manager +3. **Data Transfer**: The client directly writes data to the selected Datanodes according to the replication pipeline + +#### Read Path Sequence + +For reads, the process is simpler: + +1. The client requests key location information from the Ozone Manager +2. Using the block location information, the client reads data directly from Datanodes +3. In case of failures, the client retries with alternative replicas + +#### Monitoring and Management + + +The Recon service continuously: +- Collects metrics from the Ozone Manager, Storage Container Manager, and Datanodes +- Provides consolidated views of system health and performance +- Facilitates troubleshooting and management + +## Ozone Internals + +Understanding Ozone's internal structure helps to grasp how data is organized and protected. + +### Containers + +Containers are the fundamental storage units in Ozone: + +- Fixed-size (typically 5GB) units of storage +- Managed by the Storage Container Manager (SCM) +- Replicated or erasure-coded across datanodes +- Contain multiple blocks +- Include metadata and chunk files + +Containers are self-contained units that include all necessary metadata and data. They are the unit of replication in Ozone. + +### Blocks + +Blocks are logical units of data within containers: + +- Represent portions of objects/keys +- Created when clients write data +- Referenced by object metadata +- Allocated by the Ozone Manager +- Secured with block tokens + +When a client writes data, the OM allocates blocks from SCM, and the client writes data to these blocks through datanodes. + +### Chunks + +Chunks are the physical data units stored on disk: + +- Fixed-size portions of blocks +- Written sequentially in container data files +- Checksummed for data integrity +- Optimized for disk I/O + +Chunks are the smallest units of data stored on disk and include checksums for integrity verification. + + + +### Replicated Containers + +Ozone provides durability through container replication: + +- Default replication factor is 3 +- Uses Ratis (Raft) consensus protocol +- Synchronously replicates data across datanodes +- Provides strong consistency guarantees +- Handles node failures transparently + +Replicated containers ensure data durability by storing multiple copies of each container across different datanodes. + + + +### Erasure Encoded Containers + +Erasure coding provides space-efficient durability: + +- Splits data across multiple datanodes with parity +- Supports various coding schemes (e.g., RS-3-2-1024k) +- Reduces storage overhead compared to replication +- Trades some performance for storage efficiency +- Suitable for cold data storage + +Erasure coding allows for data durability with less storage overhead than full replication. + + + +### Pipelines + +Pipelines are groups of datanodes that work together to store data: + +- Managed by SCM +- Consist of multiple datanodes +- Handle write operations as a unit +- Support different replication strategies + +For detailed information, see [Write Pipelines](http://localhost:3001/docs/core-concepts/replication/write-pipelines). + +#### Ratis Replicated + +Ratis pipelines use the Raft consensus protocol: + +- Typically three datanodes per pipeline +- One leader and multiple followers +- Synchronous replication +- Strong consistency guarantees +- Automatic leader election on failure + +#### Erasure Coded + +Erasure coded pipelines distribute data and parity: + +- Datanodes store data or parity chunks +- EC pipeline size depends on the coding scheme +- Handles reconstruction on node failure +- Optimized for storage efficiency + +## Multi-Protocol Access + +Ozone supports multiple access protocols, making it versatile for different applications: + +### Native API (o3) + +- Command-line interface and Java API +- Full feature access +- Highest performance + +### Ozone File System (OFS) + +- Hadoop-compatible filesystem interface +- Works with all Hadoop ecosystem applications +- Path format: `ofs://om-host/volume/bucket/key` + +### S3 Compatible + +- REST API compatible with Amazon S3 +- Works with S3 clients and SDKs +- Path format: `http://s3g-host/bucket/key` + +### HttpFS + +- REST API compatible with WebHDFS +- Enables web applications to access Ozone +- Path format: `http://httpfs-host/webhdfs/v1/volume/bucket/key` + +The multi-protocol architecture allows for flexible integration with existing applications and workflows. For more information, see [Client Interfaces](../04-user-guide/01-client-interfaces/README.mdx). + +## Summary + +Apache Ozone's architecture provides: + +1. **Scalability** through separation of metadata and data paths +2. **Reliability** via replication and erasure coding +3. **Flexibility** with multiple access protocols +4. **Performance** through optimized data paths +5. **Manageability** with comprehensive monitoring + +This architecture enables Ozone to serve as both an object store and a filesystem, making it suitable for a wide range of applications from big data analytics to cloud-native workloads. + +For more detailed information about Ozone's components and internals, see the [System Internals](../07-system-internals/README.mdx) section. \ No newline at end of file diff --git a/docs/07-system-internals/01-components/07-httpfs/01-overview.md b/docs/07-system-internals/01-components/07-httpfs/01-overview.md new file mode 100644 index 00000000..92fc0aef --- /dev/null +++ b/docs/07-system-internals/01-components/07-httpfs/01-overview.md @@ -0,0 +1,79 @@ +# HttpFS Overview + +HttpFS (HTTP FileSystem) is a gateway service that provides a REST interface to Apache Ozone, compatible with the WebHDFS API. It allows web applications and non-Java clients to interact with Ozone using standard HTTP methods, without requiring the full Hadoop client library stack. + +## Role in Ozone Architecture + +In the Ozone architecture, HttpFS serves as a specialized gateway: + +- Provides HTTP/HTTPS access to Ozone storage +- Translates WebHDFS REST API calls to Ozone operations +- Acts as a proxy between web clients and Ozone components +- Offers cross-firewall access to Ozone data + +## Key Characteristics + +HttpFS provides several important capabilities: + +- **REST API Compatibility**: Implements the WebHDFS REST API, making it compatible with existing tools and applications designed for HDFS +- **HTTP/HTTPS Support**: Enables secure access through HTTPS with proper certificate configuration +- **Cross-Platform Support**: Allows non-Java clients to interact with Ozone +- **Web Application Integration**: Simplifies integration with web-based tools and services +- **Firewall Traversal**: Provides a single entry point for accessing Ozone across network boundaries + +## Internal Architecture + +Internally, HttpFS consists of several key components: + +1. **HTTP Server**: Receives and processes REST API requests +2. **Request Processors**: Handlers for different HTTP operations (GET, PUT, POST, DELETE) +3. **Authentication Filters**: SPNEGO/Kerberos and delegation token authentication +4. **Ozone Client**: A specialized Ozone client for executing operations +5. **Response Generators**: Formats responses according to WebHDFS specifications + +## Request Flow + +When HttpFS receives a request, it follows this general flow: + +1. **Authentication**: Verifies the user's credentials or delegation token +2. **Request Parsing**: Extracts the operation and parameters from the HTTP request +3. **Permission Check**: Verifies that the user has permission to perform the operation +4. **Operation Execution**: Converts the REST request to the corresponding Ozone operation and executes it +5. **Response Generation**: Creates an HTTP response with the appropriate status code and response body + +## Integration with Ozone + +HttpFS integrates with other Ozone components: + +- Communicates with the **Ozone Manager** for namespace operations +- Works with the **Storage Container Manager** for container-related operations +- Interacts directly with **Datanodes** for data transfer operations +- Supports the same **security mechanisms** as other Ozone components + +## Use Cases + +HttpFS is particularly valuable in scenarios such as: + +- Web applications that need to access Ozone data +- Cross-platform applications written in languages other than Java +- Environments where firewall constraints limit direct access to Ozone components +- Integration with existing tools that support the WebHDFS API + +## Security Considerations + +HttpFS inherits Ozone's security model: + +- **Authentication**: Supports Kerberos authentication +- **Authorization**: Respects Ozone's permission model +- **Encryption**: Supports SSL/TLS for secure communication +- **Delegation Tokens**: Allows authenticated operations without repeatedly using Kerberos credentials + +## Configuration + +HttpFS requires its own configuration, including: + +- Server port and address +- Authentication settings +- Kerberos principal and keytab (when security is enabled) +- SSL/TLS certificate details (for HTTPS) +- Connection parameters to Ozone components \ No newline at end of file diff --git a/docs/07-system-internals/01-components/07-httpfs/02-request-flow.md b/docs/07-system-internals/01-components/07-httpfs/02-request-flow.md new file mode 100644 index 00000000..9ec2fa3e --- /dev/null +++ b/docs/07-system-internals/01-components/07-httpfs/02-request-flow.md @@ -0,0 +1,122 @@ +# HttpFS Request Flow + +This document describes the journey of a client request through the HttpFS service, detailing how WebHDFS API calls are transformed into Ozone operations. + +## Request Processing Sequence + +A typical request flow through HttpFS follows these steps: + +1. **Client Request Initiation** + - Client sends an HTTP request to the HttpFS endpoint + - Request includes an operation type, path, and optional parameters + +2. **HTTP Server Processing** + - The embedded HTTP server (Jetty) receives the request + - The request is routed to the appropriate servlet based on the path + +3. **Authentication** + - Authentication filter intercepts the request + - Authenticates using one of: + - Kerberos (SPNEGO) + - Delegation token + - Simple authentication (if security is disabled) + +4. **Request Validation** + - Validates request parameters + - Checks that required parameters are present + - Verifies parameter format and values + +5. **User Resolution** + - Resolves the authenticated user's identity + - Sets up the user context for authorization + +6. **Path Resolution** + - Parses the WebHDFS path into Ozone volume, bucket, and key components + - Transforms relative paths to absolute paths if necessary + +7. **Operation Translation** + - Maps WebHDFS operations to corresponding Ozone operations: + - `CREATE` → Ozone key create + - `OPEN` → Ozone key read + - `MKDIRS` → Ozone directory creation + - `RENAME` → Ozone key rename + - And other similar mappings + +8. **Authorization Check** + - Checks if the user has permission to perform the operation + - Verifies access according to Ozone's permission model + +9. **Ozone Client Interaction** + - Creates the appropriate Ozone client call + - Executes the operation through Ozone client libraries + +10. **Data Transfer (for read/write operations)** + - For read operations: streams data from Ozone to the HTTP response + - For write operations: streams data from the HTTP request to Ozone + +11. **Response Generation** + - Creates an HTTP response with the appropriate status code + - Formats the response body according to WebHDFS specification + - Includes error details if the operation failed + +12. **Response Transmission** + - Sends the response back to the client + - Closes the connection if no more data will be exchanged + +## Specific Operation Flows + +### File Read Operation + +1. Client issues `GET /webhdfs/v1/volume/bucket/path/to/file?op=OPEN` +2. HttpFS authenticates the request +3. Translates WebHDFS path to Ozone path: `ozone://om-host/volume/bucket/path/to/file` +4. Opens input stream from Ozone client +5. Streams data through HTTP response + +### File Write Operation + +1. Client issues `PUT /webhdfs/v1/volume/bucket/path/to/file?op=CREATE` +2. HttpFS authenticates the request +3. Translates WebHDFS path to Ozone path: `ozone://om-host/volume/bucket/path/to/file` +4. Creates output stream through Ozone client +5. Reads from HTTP request body and writes to the output stream +6. Closes stream and confirms completion + +### Directory Creation + +1. Client issues `PUT /webhdfs/v1/volume/bucket/path/to/dir?op=MKDIRS` +2. HttpFS authenticates the request +3. Translates to Ozone path: `ozone://om-host/volume/bucket/path/to/dir` +4. Creates directory through Ozone client +5. Returns success status + +### Listing Directory Contents + +1. Client issues `GET /webhdfs/v1/volume/bucket/path/to/dir?op=LISTSTATUS` +2. HttpFS authenticates the request +3. Translates to Ozone path: `ozone://om-host/volume/bucket/path/to/dir` +4. Lists contents through Ozone client +5. Formats result as JSON according to WebHDFS specification +6. Returns formatted listing + +## Error Handling + +When errors occur during processing: + +1. HttpFS catches exceptions from the Ozone client +2. Maps Ozone-specific exceptions to appropriate HTTP status codes: + - `FileNotFoundException` → 404 Not Found + - `AccessControlException` → 403 Forbidden + - `InvalidPathException` → 400 Bad Request +3. Generates a JSON response with error details +4. Logs the error with appropriate severity + +## Security Context Flow + +For secure clusters, the security context flows as follows: + +1. Client authenticates using Kerberos or delegation token +2. HttpFS validates credentials and creates a user proxy +3. All Ozone operations are executed as the authenticated user +4. Authorization checks are performed against the user's identity and groups +5. Audit logs record the original user's actions \ No newline at end of file diff --git a/docs/07-system-internals/01-components/07-httpfs/README.mdx b/docs/07-system-internals/01-components/07-httpfs/README.mdx new file mode 100644 index 00000000..d6c564c4 --- /dev/null +++ b/docs/07-system-internals/01-components/07-httpfs/README.mdx @@ -0,0 +1,11 @@ +--- +sidebar_label: HttpFS +--- + +# HttpFS Service + +import DocCardList from '@theme/DocCardList'; + +HttpFS is a specialized gateway service that provides WebHDFS-compatible REST API access to Apache Ozone. It serves as a bridge between web applications and Ozone storage, enabling them to use the familiar WebHDFS interface for operations. + +<DocCardList/> \ No newline at end of file diff --git a/src/css/custom.css b/src/css/custom.css index 7c7f82b0..6601470b 100644 --- a/src/css/custom.css +++ b/src/css/custom.css @@ -78,6 +78,56 @@ h1, h2, h3, h4, h5, h6 { letter-spacing: -0.02em; } +/* Colored headings for markdown */ +.markdown h1 { + color: var(--ifm-color-primary-darkest); +} + +.markdown h2 { + color: var(--ifm-color-primary-darker); +} + +.markdown h3 { + color: var(--ifm-color-primary-dark); +} + +.markdown h4 { + color: var(--ifm-color-primary); +} + +.markdown h5 { + color: var(--ifm-color-primary-light); +} + +.markdown h6 { + color: var(--ifm-color-primary-lighter); +} + +/* Dark mode heading colors */ +[data-theme='dark'] .markdown h1 { + color: var(--ifm-color-primary-lightest); +} + +[data-theme='dark'] .markdown h2 { + color: var(--ifm-color-primary-lighter); +} + +[data-theme='dark'] .markdown h3 { + color: var(--ifm-color-primary-light); +} + +[data-theme='dark'] .markdown h4 { + color: var(--ifm-color-primary); +} + +[data-theme='dark'] .markdown h5 { + color: var(--ifm-color-primary-dark); +} + +[data-theme='dark'] .markdown h6 { + color: var(--ifm-color-primary-darker); +} + pre, code { font-family: var(--ifm-font-family-monospace); font-size: 14px; diff --git a/src/pages/community/ask-a-question.md b/src/pages/community/ask-a-question.md index 28eb1b2f..ce45c267 100644 --- a/src/pages/community/ask-a-question.md +++ b/src/pages/community/ask-a-question.md @@ -3,7 +3,10 @@ <!-- markdownlint-disable --> import AskQuestionForm from '@site/src/components/AskQuestionForm'; -Have a question about Apache Ozone? Use the form below to submit your question to our GitHub Discussions. The form will redirect you to GitHub with your question pre-filled. +Have a question about Apache Ozone? +Use the form below to submit your question to our GitHub Discussions. +The form will redirect you to GitHub with your question pre-filled +or [you can head to posting a new topic on the FAQ section](https://github.com/apache/ozone/discussions/new?category=faq) <AskQuestionForm /> <!-- markdownlint-enable --> diff --git a/static/img/ozone/container.svg b/static/img/ozone/container.svg new file mode 100644 index 00000000..a0d86753 --- /dev/null +++ b/static/img/ozone/container.svg @@ -0,0 +1,173 @@ +<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1000 730"> + <!-- Background --> + <rect width="1000" height="850" fill="transparent" /> + + <!-- Title --> + <text x="500" y="40" font-family="Inter, sans-serif" font-size="26" text-anchor="middle" fill="#357500" font-weight="bold">Apache Ozone Storage Hierarchy</text> + + <!-- Datanode --> + <rect x="50" y="70" width="900" height="630" rx="15" ry="15" fill="#e0edd2" stroke="#357500" stroke-width="4" /> + <text x="500" y="110" font-family="Inter, sans-serif" font-size="30" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-42</text> + + <!-- Disk-Volumes --> + <rect x="80" y="120" width="410" height="550" rx="12" ry="12" fill="#d1e4bc" stroke="#2d6300" stroke-width="3.5" /> + <text x="285" y="155" font-family="Inter, sans-serif" font-size="26" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-107</text> + + <rect x="510" y="120" width="410" height="550" rx="12" ry="12" fill="#d1e4bc" stroke="#2d6300" stroke-width="3.5" /> + <text x="715" y="155" font-family="Inter, sans-serif" font-size="26" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-394</text> + + <!-- Containers in Disk-Volume 1 --> + <rect x="100" y="180" width="370" height="230" rx="10" ry="10" fill="#96d26e" stroke="#3a8100" stroke-width="3" /> + <text x="285" y="210" font-family="Inter, sans-serif" font-size="24" text-anchor="middle" fill="#255200" font-weight="bold">Container-583</text> + + <rect x="100" y="430" width="370" height="220" rx="10" ry="10" fill="#96d26e" stroke="#3a8100" stroke-width="3" /> + <text x="285" y="460" font-family="Inter, sans-serif" font-size="24" text-anchor="middle" fill="#255200" font-weight="bold">Container-217</text> + + <!-- Containers in Disk-Volume 2 --> + <rect x="530" y="180" width="370" height="230" rx="10" ry="10" fill="#96d26e" stroke="#3a8100" stroke-width="3" /> + <text x="715" y="210" font-family="Inter, sans-serif" font-size="24" text-anchor="middle" fill="#255200" font-weight="bold">Container-965</text> + + <rect x="530" y="430" width="370" height="220" rx="10" ry="10" fill="#96d26e" stroke="#3a8100" stroke-width="3" /> + <text x="715" y="460" font-family="Inter, sans-serif" font-size="24" text-anchor="middle" fill="#255200" font-weight="bold">Container-742</text> + + <!-- Blocks in Container 1 --> + <rect x="120" y="240" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="195" y="265" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-84</text> + <text x="195" y="290" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 235)</text> + + <rect x="300" y="240" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="375" y="265" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-129</text> + <text x="375" y="290" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 517)</text> + + <!-- Blocks in Container 2 --> + <rect x="120" y="480" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="195" y="505" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-352</text> + <text x="195" y="530" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 673)</text> + + <rect x="300" y="480" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="375" y="505" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-198</text> + <text x="375" y="530" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 429)</text> + + <!-- Blocks in Container 3 --> + <rect x="550" y="240" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="625" y="265" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-456</text> + <text x="625" y="290" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 814)</text> + + <rect x="730" y="240" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="805" y="265" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-762</text> + <text x="805" y="290" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 329)</text> + + <!-- Blocks in Container 4 --> + <rect x="550" y="480" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="625" y="505" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-507</text> + <text x="625" y="530" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 946)</text> + + <rect x="730" y="480" width="150" height="155" rx="8" ry="8" fill="#64ab35" stroke="#306900" stroke-width="2.5" /> + <text x="805" y="505" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-631</text> + <text x="805" y="530" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#ffffff">(Object 182)</text> + + <!-- Chunks in Block-84 - wider chunks --> + <rect x="130" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="162" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-37</text> + + <rect x="130" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="162" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-92</text> + + <rect x="200" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="232" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-15</text> + + <rect x="200" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="232" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-64</text> + + <!-- Chunks in Block-129 --> + <rect x="310" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="342" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-23</text> + + <rect x="310" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="342" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-51</text> + + <rect x="380" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="412" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-88</text> + + <rect x="380" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="412" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-47</text> + + <!-- Chunks in Block-352 --> + <rect x="130" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="162" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-74</text> + + <rect x="130" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="162" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-19</text> + + <rect x="200" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="232" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-83</text> + + <rect x="200" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="232" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-42</text> + + <!-- Chunks in Block-198 --> + <rect x="310" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="342" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-67</text> + + <rect x="310" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="342" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-29</text> + + <rect x="380" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="412" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-54</text> + + <rect x="380" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="412" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-78</text> + + <!-- Chunks in Block-456 --> + <rect x="560" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="592" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-11</text> + + <rect x="560" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="592" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-95</text> + + <rect x="630" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="662" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-30</text> + + <rect x="630" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="662" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-86</text> + + <!-- Chunks in Block-762 --> + <rect x="740" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="772" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-43</text> + + <rect x="740" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="772" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-68</text> + + <rect x="810" y="305" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="842" y="330" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-21</text> + + <rect x="810" y="350" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="842" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-59</text> + + <!-- Chunks in Block-507 --> + <rect x="560" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="592" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-32</text> + + <rect x="560" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="592" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-75</text> + + <rect x="630" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="662" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-27</text> + + <rect x="630" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="662" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-90</text> + + <!-- Chunks in Block-631 --> + <rect x="740" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="772" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-56</text> + + <rect x="740" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="772" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-13</text> + + <rect x="810" y="545" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="842" y="570" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-81</text> + + <rect x="810" y="590" width="65" height="40" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="842" y="615" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="middle" fill="white" font-weight="bold">Chunk-49</text> + +</svg> diff --git a/static/img/ozone/ozone-client-interaction.svg b/static/img/ozone/ozone-client-interaction.svg new file mode 100644 index 00000000..39ac95ba --- /dev/null +++ b/static/img/ozone/ozone-client-interaction.svg @@ -0,0 +1,209 @@ +<svg width="1400" height="700" xmlns="http://www.w3.org/2000/svg"; font-family="JetBrains Mono, monospace" font-size="12"> + <!-- Background --> + <rect width="1400" height="700" fill="#f7fcf0" /> + + <!-- Title --> + <text x="700" y="50" font-family="JetBrains Mono, monospace" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Apache Ozone Client Interactions</text> + + <defs> + <!-- Arrowhead marker definition --> + <marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"> + <path d="M 0 0 L 10 5 L 0 10 z" fill="#255200" /> + </marker> + </defs> + + <style> + .user-icon { stroke: #255200; stroke-width: 2; fill: none; } + .interface-box { fill: #d1e4bc; stroke: #3a8100; stroke-width: 2; rx: 8; ry: 8; } + .interface-text { fill: #255200; text-anchor: middle; font-weight: bold; } + /* Updated enclosing box style */ + .enclosing-box { fill: none; stroke: #2d6300; stroke-width: 1.5; stroke-dasharray: 6 4; rx: 10; ry: 10; } + .client-box { fill: #96d26e; stroke: #3a8100; stroke-width: 2.5; rx: 8; ry: 8; } + /* Updated client text style */ + .client-text { fill: #255200; text-anchor: middle; font-weight: bold; } + .arrow-line { stroke: #255200; stroke-width: 1.5; marker-end: url(#arrow); } + /* Ozone Manager style - Updated color */ + .manager-box { fill: #4c8c2b; stroke: #306900; stroke-width: 2.5; rx: 8; ry: 8; } + .manager-text { fill: #ffffff; text-anchor: middle; font-weight: bold; } + /* SCM style - New color */ + .scm-box { fill: #3a7a1a; stroke: #306900; stroke-width: 2.5; rx: 8; ry: 8; } + /* Datanode style - New color */ + .datanode-box { fill: #5fa638; stroke: #3a8100; stroke-width: 2; rx: 8; ry: 8; } + </style> + + <!-- Data Enclosing Box --> + <rect x="245" y="380" width="770" height="300" rx="15" ry="15" stroke="#357500" stroke-width="2" fill="#e0edd2"/> + <text x="630" y="400" font-family="JetBrains Mono, monospace" font-size="18" text-anchor="middle" fill="#357500" font-weight="bold">Data</text> + + <!-- User Stick Figure --> + <g transform="translate(50, 195)"> + <circle cx="0" cy="-20" r="10" class="user-icon" fill="#64ab35"/> + <line x1="0" y1="-10" x2="0" y2="10" class="user-icon"/> + <line x1="-15" y1="0" x2="15" y2="0" class="user-icon"/> + <line x1="0" y1="10" x2="-10" y2="30" class="user-icon"/> + <line x1="0" y1="10" x2="10" y2="30" class="user-icon"/> + <text x="0" y="45" text-anchor="middle" font-size="16" fill="#255200" font-weight="bold" font-family="JetBrains Mono, monospace">User</text> + </g> + + <!-- Enclosing Box for Interfaces AND Client Library --> + <rect x="140" y="80" width="310" height="270" rx="15" ry="15" stroke="#357500" stroke-width="2" fill="#e0edd2"/> + <text x="295" y="105" font-family="JetBrains Mono, monospace" font-size="18" text-anchor="middle" fill="#357500" font-weight="bold">Client Application</text> + + <!-- Interface/Application Boxes --> + <g id="interfaces"> + <rect x="150" y="120" width="120" height="30" class="interface-box"/> + <text x="210" y="140" class="interface-text" font-size="16" font-family="JetBrains Mono, monospace">S3 Gateway</text> + + <rect x="150" y="160" width="120" height="30" class="interface-box"/> + <text x="210" y="180" class="interface-text" font-size="16" font-family="JetBrains Mono, monospace">HttpFS</text> + + <rect x="150" y="200" width="120" height="30" class="interface-box"/> + <text x="210" y="220" class="interface-text" font-size="16" font-family="JetBrains Mono, monospace">Hive</text> + + <rect x="150" y="240" width="120" height="30" class="interface-box"/> + <text x="210" y="260" class="interface-text" font-size="16" font-family="JetBrains Mono, monospace">Spark</text> + + <rect x="150" y="280" width="120" height="30" class="interface-box"/> + <text x="210" y="300" class="interface-text" font-size="16" font-family="JetBrains Mono, monospace">Iceberg</text> + </g> + + <!-- Ozone Client Library Box --> + <rect x="320" y="120" width="120" height="210" class="client-box"/> + <!-- Updated Text --> + <text x="380" y="210" class="client-text" font-size="16" font-family="JetBrains Mono, monospace">Ozone</text> + <text x="380" y="230" class="client-text" font-size="16" font-family="JetBrains Mono, monospace">Client</text> + <text x="380" y="250" class="client-text" font-size="16" font-family="JetBrains Mono, monospace">Library</text> + + <!-- Metadata Enclosing Box --> + <rect x="540" y="70" width="650" height="220" rx="15" ry="15" stroke="#357500" stroke-width="2" fill="#e0edd2"/> + <text x="865" y="90" font-family="JetBrains Mono, monospace" font-size="18" text-anchor="middle" fill="#357500" font-weight="bold">Metadata</text> + + <!-- OM Ratis Components --> + <rect x="550" y="95" width="290" height="180" fill="#d1e4bc" stroke="#2d6300" stroke-width="1.5" rx="10" ry="10"/> + <text x="695" y="115" text-anchor="middle" font-weight="bold" font-size="16" fill="#306900" font-family="JetBrains Mono, monospace">Ratis (Raft)</text> + + <!-- Leader Manager Box --> + <rect x="560" y="130" width="270" height="50" class="manager-box"/> + <text x="695" y="160" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">OM (Leader)</text> + + <!-- Follower Boxes Row --> + <g id="om-followers-row"> + <!-- Follower 1 Manager Box --> + <rect x="560" y="190" width="130" height="50" class="manager-box" fill-opacity="0.85"/> + <text x="625" y="220" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">OM (Follower)</text> + + <!-- Follower 2 Manager Box --> + <rect x="700" y="190" width="130" height="50" class="manager-box" fill-opacity="0.85"/> + <text x="765" y="220" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">OM (Follower)</text> + </g> + + <!-- Storage Container Manager Boxes - Ratis(Raft) --> + <g id="storage-container-managers"> + <!-- Enclosing box for SCM Ratis group --> + <rect x="550" y="95" width="290" height="180" fill="#d1e4bc" stroke="#2d6300" stroke-width="1.5" rx="10" ry="10" transform="translate(340, 0)"/> + <text x="695" y="115" text-anchor="middle" font-weight="bold" font-size="16" fill="#306900" font-family="JetBrains Mono, monospace" transform="translate(340, 0)">Ratis (Raft)</text> + + <!-- Leader Storage Container Manager Box --> + <rect x="560" y="130" width="270" height="50" class="scm-box" transform="translate(340, 0)"/> + <text x="695" y="160" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace" transform="translate(340, 0)">SCM (Leader)</text> + + <!-- Follower Boxes Row --> + <g id="scm-followers-row" transform="translate(340, 0)"> + <!-- Follower 1 Storage Container Manager Box --> + <rect x="560" y="190" width="130" height="50" class="scm-box" fill-opacity="0.85"/> + <text x="625" y="220" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">SCM (Follower)</text> + + <!-- Follower 2 Storage Container Manager Box --> + <rect x="700" y="190" width="130" height="50" class="scm-box" fill-opacity="0.85"/> + <text x="765" y="220" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">SCM (Follower)</text> + </g> + </g> + + <!-- Arrows --> + <!-- User -> Interfaces (Simplified to point to the enclosing box area) --> + <line x1="80" y1="195" x2="140" y2="195" class="arrow-line"/> + + <!-- Interfaces -> Ozone Client Library --> + <line x1="270" y1="135" x2="320" y2="135" class="arrow-line"/> <!-- S3 GW -> Client --> + <line x1="270" y1="175" x2="320" y2="175" class="arrow-line"/> <!-- HttpFS -> Client --> + <line x1="270" y1="215" x2="320" y2="215" class="arrow-line"/> <!-- Hive -> Client --> + <line x1="270" y1="255" x2="320" y2="255" class="arrow-line"/> <!-- Spark -> Client --> + <line x1="270" y1="295" x2="320" y2="295" class="arrow-line"/> <!-- Iceberg -> Client --> + + <!-- Ozone Client Library -> Ozone Manager Leader --> + <line x1="440" y1="160" x2="560" y2="160" class="arrow-line"/> <!-- Client -> Leader Manager --> + <text x="460" y="145" font-size="12" font-family="JetBrains Mono, monospace" fill="#255200" font-weight="bold">Metadata</text> + <text x="460" y="175" font-size="12" font-family="JetBrains Mono, monospace" fill="#255200" font-weight="bold">Operations</text> + + + <!-- Text for OM -> SCM communication --> + + <!-- Datanode Boxes: 3x5 Grid --> + <!-- Row 1 --> + <rect x="265" y="420" width="130" height="60" class="datanode-box"/> + <text x="330" y="455" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="415" y="420" width="130" height="60" class="datanode-box"/> + <text x="480" y="455" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="565" y="420" width="130" height="60" class="datanode-box"/> + <text x="630" y="455" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="715" y="420" width="130" height="60" class="datanode-box"/> + <text x="780" y="455" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="865" y="420" width="130" height="60" class="datanode-box"/> + <text x="930" y="455" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <!-- Row 2 --> + <rect x="265" y="500" width="130" height="60" class="datanode-box"/> + <text x="330" y="535" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="415" y="500" width="130" height="60" class="datanode-box"/> + <text x="480" y="535" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="565" y="500" width="130" height="60" class="datanode-box"/> + <text x="630" y="535" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="715" y="500" width="130" height="60" class="datanode-box"/> + <text x="780" y="535" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="865" y="500" width="130" height="60" class="datanode-box"/> + <text x="930" y="535" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <!-- Row 3 --> + <rect x="265" y="580" width="130" height="60" class="datanode-box"/> + <text x="330" y="615" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="415" y="580" width="130" height="60" class="datanode-box"/> + <text x="480" y="615" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="565" y="580" width="130" height="60" class="datanode-box"/> + <text x="630" y="615" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="715" y="580" width="130" height="60" class="datanode-box"/> + <text x="780" y="615" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="865" y="580" width="130" height="60" class="datanode-box"/> + <text x="930" y="615" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <!-- Curved Arrow from Data to SCM Leader following right edge then up --> + <path d="M 1015 430 L 1245 430 L 1245 155 L 1170 155" + stroke="#255200" stroke-width="2.5" stroke-dasharray="5,3" fill="none" marker-end="url(#arrow)"/> <!-- Data -> SCM Leader --> + <text x="1195" y="135" font-size="12" font-family="JetBrains Mono, monospace" fill="#255200" font-weight="bold">Heartbeats and Reports</text> + + <!-- Vertical Arrow from Client to Data --> + <path d="M 380 330 L 380 380" stroke="#255200" stroke-width="2.5" stroke-dasharray="5,3" marker-end="url(#arrow)"/> <!-- Client -> Data --> + <text x="460" y="370" font-size="12" font-family="JetBrains Mono, monospace" fill="#255200" font-weight="bold">I/O Operations</text> + + <!-- Replication arrows from Leader to Followers --> + <path d="M 695 180 L 625 190" stroke="#306900" stroke-width="1.5" marker-end="url(#arrow)"/> <!-- Leader -> Follower 1 --> + <path d="M 695 180 L 765 190" stroke="#306900" stroke-width="1.5" marker-end="url(#arrow)"/> <!-- Leader -> Follower 2 --> + + <!-- Replication arrows from SCM Leader to Followers --> + <path d="M 1035 180 L 965 190" stroke="#306900" stroke-width="1.5" marker-end="url(#arrow)"/> <!-- SCM Leader -> Follower 1 --> + <path d="M 1035 180 L 1105 190" stroke="#306900" stroke-width="1.5" marker-end="url(#arrow)"/> <!-- SCM Leader -> Follower 2 --> + + <!-- Communication arrow between Ozone Manager and Storage Container Manager --> + <path d="M 830 155 L 900 155" stroke="#255200" stroke-width="2.5" stroke-dasharray="5,3" marker-end="url(#arrow)"/> <!-- OM Leader -> SCM Leader --> +</svg> \ No newline at end of file diff --git a/static/img/ozone/ozone-namespace.svg b/static/img/ozone/ozone-namespace.svg new file mode 100644 index 00000000..a7f2171b --- /dev/null +++ b/static/img/ozone/ozone-namespace.svg @@ -0,0 +1,79 @@ +<?xml version="1.0" encoding="UTF-8"?> +<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1000 700"> + <!-- Background --> + <rect width="1000" height="700" fill="transparent" /> + + <!-- Title --> + <text x="500" y="40" font-family="Inter, sans-serif" font-size="26" text-anchor="middle" fill="#357500" font-weight="bold">Apache Ozone Namespace Hierarchy</text> + + <!-- Ozone cluster --> + <rect x="100" y="90" width="800" height="580" rx="10" ry="10" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="500" y="120" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Ozone Namespace</text> + + <!-- Volume 1 - Left side --> + <rect x="150" y="150" width="320" height="480" rx="8" ry="8" fill="#d1e4bc" stroke="#2d6300" stroke-width="2.5" /> + <text x="310" y="180" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#2d6300" font-weight="bold">Volume 1</text> + <text x="310" y="205" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#2d6300">(tenant1)</text> + + <!-- Volume 2 - Right side --> + <rect x="530" y="150" width="320" height="480" rx="8" ry="8" fill="#d1e4bc" stroke="#2d6300" stroke-width="2.5" /> + <text x="690" y="180" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#2d6300" font-weight="bold">Volume 2</text> + <text x="690" y="205" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#2d6300">(tenant2)</text> + + <!-- Buckets in Volume 1 --> + <rect x="170" y="230" width="280" height="180" rx="6" ry="6" fill="#96d26e" stroke="#3a8100" stroke-width="2" /> + <text x="310" y="255" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#255200" font-weight="bold">Bucket A</text> + <text x="310" y="275" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#255200">(FSO Layout)</text> + + <rect x="170" y="430" width="280" height="180" rx="6" ry="6" fill="#96d26e" stroke="#3a8100" stroke-width="2" /> + <text x="310" y="455" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#255200" font-weight="bold">Bucket B</text> + <text x="310" y="475" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#255200">(OBS Layout)</text> + + <!-- Keys/Objects in Bucket A (Folder structure) --> + <rect x="190" y="295" width="240" height="105" rx="4" ry="4" fill="#64ab35" stroke="#306900" stroke-width="1.5" /> + <text x="310" y="315" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Directory Structure</text> + + <!-- Directory tree --> + <text x="210" y="335" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">/ (root)</text> + <text x="225" y="355" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">├── dir1/</text> + <text x="240" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ ├── file1.txt</text> + <text x="240" y="395" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ └── file2.txt</text> + + <!-- Keys/Objects in Bucket B (Flat namespace) --> + <rect x="190" y="495" width="240" height="105" rx="4" ry="4" fill="#64ab35" stroke="#306900" stroke-width="1.5" /> + <text x="310" y="515" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Objects (Flat)</text> + + <!-- Object list --> + <text x="215" y="535" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">object1</text> + <text x="215" y="555" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">object2</text> + <text x="215" y="575" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">object3</text> + <text x="215" y="595" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">object4</text> + + <!-- Buckets in Volume 2 (the encrypted bucket with deep structure) --> + <rect x="550" y="230" width="280" height="380" rx="6" ry="6" fill="#96d26e" stroke="#3a8100" stroke-width="2" /> + <text x="690" y="255" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#255200" font-weight="bold">Bucket F</text> + <text x="690" y="275" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#255200">(FSO Layout + Encryption)</text> + + <!-- Security icon for encrypted bucket --> + <circle cx="610" y="255" r="10" fill="#ffffff" stroke="#255200" stroke-width="1" /> + <text x="610" y="260" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200" font-weight="bold">🔒</text> + + <!-- Directory structure in Bucket F - Deep Structure --> + <rect x="570" y="295" width="240" height="305" rx="4" ry="4" fill="#64ab35" stroke="#306900" stroke-width="1.5" /> + <text x="690" y="315" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Directory Structure</text> + + <!-- Deep directory tree --> + <text x="590" y="335" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">/ (root)</text> + <text x="590" y="355" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">├── data/</text> + <text x="590" y="375" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ ├── logs/</text> + <text x="590" y="395" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ │ ├── 2023/</text> + <text x="590" y="415" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ │ └── 2024/</text> + <text x="590" y="435" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ └── backup/</text> + <text x="590" y="455" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">├── apps/</text> + <text x="590" y="475" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ ├── app1/</text> + <text x="590" y="495" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">│ └── app2/</text> + <text x="590" y="515" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff">└── config/</text> + <text x="590" y="535" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff"> ├── dev/</text> + <text x="590" y="555" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff"> ├── staging/</text> + <text x="590" y="575" font-family="JetBrains Mono, monospace" font-size="12" text-anchor="start" fill="#ffffff"> └── prod/</text> +</svg> \ No newline at end of file diff --git a/static/img/ozone/ozone-storage-hierarchy-ec.svg b/static/img/ozone/ozone-storage-hierarchy-ec.svg new file mode 100644 index 00000000..0cf7c326 --- /dev/null +++ b/static/img/ozone/ozone-storage-hierarchy-ec.svg @@ -0,0 +1,142 @@ +<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1100 900"> + <!-- Background --> + <rect width="1100" height="900" fill="transparent" /> + + <!-- Title with improved styling --> + <text x="550" y="40" font-family="Inter, sans-serif" font-size="26" text-anchor="middle" fill="#357500" font-weight="bold">Apache Ozone Erasure Coding (3:2 scheme)</text> + + <!-- Object Split Text - Moved to top --> + <text x="550" y="70" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#357500" font-weight="bold">Object Data Split</text> + <text x="550" y="90" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#357500">Across 3 Data Blocks</text> + + <!-- DATA NODES - First row (data blocks) --> + <!-- Datanode 1 --> + <rect x="150" y="110" width="250" height="330" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="275" y="145" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-42</text> + + <!-- Disk-Volume in Datanode 1 --> + <rect x="170" y="170" width="210" height="250" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="275" y="200" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-107</text> + + <!-- Data Container in Datanode 1 --> + <rect x="190" y="220" width="170" height="180" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="3" /> + <text x="275" y="250" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">EC Container-583</text> + <text x="275" y="270" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Data Block 1)</text> + + <!-- Block in Container --> + <rect x="210" y="290" width="130" height="90" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="275" y="312" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-84</text> + <text x="275" y="332" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 235)</text> + + <!-- Chunks in Block --> + <rect x="220" y="350" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="245" y="365" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-1</text> + + <rect x="280" y="350" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="305" y="365" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-2</text> + + <!-- Datanode 2 --> + <rect x="425" y="110" width="250" height="330" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="550" y="145" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-78</text> + + <!-- Disk-Volume in Datanode 2 --> + <rect x="445" y="170" width="210" height="250" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="550" y="200" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-254</text> + + <!-- Data Container in Datanode 2 --> + <rect x="465" y="220" width="170" height="180" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="3" /> + <text x="550" y="250" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">EC Container-584</text> + <text x="550" y="270" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Data Block 2)</text> + + <!-- Block in Container --> + <rect x="485" y="290" width="130" height="90" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="550" y="312" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-85</text> + <text x="550" y="332" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 235)</text> + + <!-- Chunks in Block --> + <rect x="495" y="350" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="520" y="365" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-1</text> + + <rect x="555" y="350" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="580" y="365" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-2</text> + + <!-- Datanode 3 --> + <rect x="700" y="110" width="250" height="330" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="825" y="145" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-96</text> + + <!-- Disk-Volume in Datanode 3 --> + <rect x="720" y="170" width="210" height="250" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="825" y="200" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-487</text> + + <!-- Data Container in Datanode 3 --> + <rect x="740" y="220" width="170" height="180" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="3" /> + <text x="825" y="250" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">EC Container-585</text> + <text x="825" y="270" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Data Block 3)</text> + + <!-- Block in Container --> + <rect x="760" y="290" width="130" height="90" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="825" y="312" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-86</text> + <text x="825" y="332" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 235)</text> + + <!-- Chunks in Block --> + <rect x="770" y="350" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="795" y="365" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-1</text> + + <rect x="830" y="350" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="855" y="365" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-2</text> + + <!-- Parity Calculation Text - Between data and parity containers --> + <text x="550" y="500" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#357500" font-weight="bold">Parity Calculation</text> + <text x="550" y="520" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#357500">P, Q = f(Data Blocks)</text> + + <!-- DATA NODES - Second row (parity blocks) --> + <!-- Datanode 4 --> + <rect x="275" y="550" width="250" height="330" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="400" y="585" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-112</text> + + <!-- Disk-Volume in Datanode 4 --> + <rect x="295" y="610" width="210" height="250" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="400" y="640" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-632</text> + + <!-- Parity Container in Datanode 4 --> + <rect x="315" y="660" width="170" height="180" rx="8" ry="8" fill="#a8da86" stroke="#3a8100" stroke-width="3" /> + <text x="400" y="690" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">EC Container-586</text> + <text x="400" y="710" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Parity Block 1)</text> + + <!-- Block in Container --> + <rect x="335" y="730" width="130" height="90" rx="6" ry="6" fill="#84ca56" stroke="#306900" stroke-width="2" /> + <text x="400" y="752" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-87</text> + <text x="400" y="772" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Parity Data P)</text> + + <!-- Chunks in Block --> + <rect x="345" y="790" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#306900" stroke-width="1.5" /> + <text x="370" y="805" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-1</text> + + <rect x="405" y="790" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#306900" stroke-width="1.5" /> + <text x="430" y="805" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-2</text> + + <!-- Datanode 5 --> + <rect x="575" y="550" width="250" height="330" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="700" y="585" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-143</text> + + <!-- Disk-Volume in Datanode 5 --> + <rect x="595" y="610" width="210" height="250" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="700" y="640" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-839</text> + + <!-- Parity Container in Datanode 5 --> + <rect x="615" y="660" width="170" height="180" rx="8" ry="8" fill="#a8da86" stroke="#3a8100" stroke-width="3" /> + <text x="700" y="690" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">EC Container-587</text> + <text x="700" y="710" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Parity Block 2)</text> + + <!-- Block in Container --> + <rect x="635" y="730" width="130" height="90" rx="6" ry="6" fill="#84ca56" stroke="#306900" stroke-width="2" /> + <text x="700" y="752" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-88</text> + <text x="700" y="772" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Parity Data Q)</text> + + <!-- Chunks in Block --> + <rect x="645" y="790" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#306900" stroke-width="1.5" /> + <text x="670" y="805" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-1</text> + + <rect x="705" y="790" width="50" height="20" rx="4" ry="4" fill="#459800" stroke="#306900" stroke-width="1.5" /> + <text x="730" y="805" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Chunk-2</text> +</svg> diff --git a/static/img/ozone/ozone-storage-hierarchy-replicated.svg b/static/img/ozone/ozone-storage-hierarchy-replicated.svg new file mode 100644 index 00000000..6555fdb9 --- /dev/null +++ b/static/img/ozone/ozone-storage-hierarchy-replicated.svg @@ -0,0 +1,281 @@ +<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1100 690"> + <!-- Improved background with softer color --> + <rect width="1100" height="860" fill="transparent" /> + + <!-- Title with improved styling --> + <text x="550" y="40" font-family="Inter, sans-serif" font-size="26" text-anchor="middle" fill="#357500" font-weight="bold">Apache Ozone Container Replication</text> + + <!-- DATANODE 1 --> + <!-- Datanode box --> + <rect x="50" y="80" width="320" height="600" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="210" y="115" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-42</text> + + <!-- Disk-Volume in Datanode 1 --> + <rect x="70" y="140" width="280" height="520" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="210" y="170" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-107</text> + + <!-- Container 1 in Datanode 1 (Replicated) --> + <rect x="90" y="190" width="240" height="220" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="3" stroke-dasharray="5,3" /> + <text x="210" y="220" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">Container-583</text> + <text x="210" y="240" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Replicated across 3 DNs)</text> + + <!-- Blocks in Container 1 of Datanode 1 --> + <rect x="110" y="260" width="90" height="130" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="155" y="282" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-84</text> + <text x="155" y="302" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 235)</text> + + <rect x="220" y="260" width="90" height="130" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="265" y="282" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-129</text> + <text x="265" y="302" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 517)</text> + + <!-- Chunks in Block-84 --> + <rect x="120" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="137" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-37</text> + + <rect x="120" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="137" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-92</text> + + <rect x="160" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="177" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-15</text> + + <rect x="160" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="177" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-64</text> + + <!-- Chunks in Block-129 --> + <rect x="230" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="247" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-23</text> + + <rect x="230" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="247" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-51</text> + + <rect x="270" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="287" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-88</text> + + <rect x="270" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="287" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-47</text> + + <!-- Container 2 in Datanode 1 (Not Replicated) --> + <rect x="90" y="430" width="240" height="210" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="2" /> + <text x="210" y="460" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#255200" font-weight="bold">Container-217</text> + + <!-- Blocks in Container 2 of Datanode 1 --> + <rect x="110" y="480" width="90" height="140" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="155" y="505" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-352</text> + <text x="155" y="525" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 673)</text> + + <!-- Chunks in Block-352 --> + <rect x="120" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="137" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-74</text> + + <rect x="120" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="137" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-19</text> + + <rect x="160" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="177" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-83</text> + + <rect x="160" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="177" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-42</text> + + <rect x="220" y="480" width="90" height="140" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="265" y="505" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-198</text> + <text x="265" y="525" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 429)</text> + + <!-- Chunks in Block-198 --> + <rect x="230" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="247" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-67</text> + + <rect x="230" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="247" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-29</text> + + <rect x="270" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="287" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-54</text> + + <rect x="270" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="287" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-78</text> + + <!-- DATANODE 2 --> + <!-- Datanode box --> + <rect x="390" y="80" width="320" height="600" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="550" y="115" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-78</text> + + <!-- Disk-Volume in Datanode 2 --> + <rect x="410" y="140" width="280" height="520" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="550" y="170" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-254</text> + + <!-- Container 1 in Datanode 2 (Replicated) --> + <rect x="430" y="190" width="240" height="220" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="3" stroke-dasharray="5,3" /> + <text x="550" y="220" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">Container-583</text> + <text x="550" y="240" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Replicated across 3 DNs)</text> + + <!-- Blocks in Container 1 of Datanode 2 --> + <rect x="450" y="260" width="90" height="130" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="495" y="282" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-84</text> + <text x="495" y="302" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 235)</text> + + <!-- Chunks in Block-84 (DN 2) --> + <rect x="460" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="477" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-37</text> + + <rect x="460" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="477" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-92</text> + + <rect x="500" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="517" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-15</text> + + <rect x="500" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="517" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-64</text> + + <rect x="560" y="260" width="90" height="130" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="605" y="282" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-129</text> + <text x="605" y="302" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 517)</text> + + <!-- Chunks in Block-129 (DN 2) --> + <rect x="570" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="587" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-23</text> + + <rect x="570" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="587" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-51</text> + + <rect x="610" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="627" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-88</text> + + <rect x="610" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="627" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-47</text> + + <!-- Container 3 in Datanode 2 (Not Replicated) --> + <rect x="430" y="430" width="240" height="210" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="2" /> + <text x="550" y="460" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#255200" font-weight="bold">Container-468</text> + + <!-- Blocks in Container 3 of Datanode 2 --> + <rect x="450" y="480" width="90" height="140" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="495" y="505" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-123</text> + <text x="495" y="525" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 321)</text> + + <!-- Chunks in Block-123 --> + <rect x="460" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="477" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-31</text> + + <rect x="460" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="477" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-62</text> + + <rect x="500" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="517" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-08</text> + + <rect x="500" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="517" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-45</text> + + <rect x="560" y="480" width="90" height="140" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="605" y="505" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-777</text> + <text x="605" y="525" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 159)</text> + + <!-- Chunks in Block-777 --> + <rect x="570" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="587" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-17</text> + + <rect x="570" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="587" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-38</text> + + <rect x="610" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="627" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-75</text> + + <rect x="610" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="627" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-93</text> + + <!-- DATANODE 3 --> + <!-- Datanode box --> + <rect x="730" y="80" width="320" height="600" rx="12" ry="12" fill="#e0edd2" stroke="#357500" stroke-width="3" /> + <text x="890" y="115" font-family="Inter, sans-serif" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Datanode-96</text> + + <!-- Disk-Volume in Datanode 3 --> + <rect x="750" y="140" width="280" height="520" rx="10" ry="10" fill="#d1e4bc" stroke="#2d6300" stroke-width="2" /> + <text x="890" y="170" font-family="Inter, sans-serif" font-size="20" text-anchor="middle" fill="#2d6300" font-weight="bold">Disk-Volume-487</text> + + <!-- Container 1 in Datanode 3 (Replicated) --> + <rect x="770" y="190" width="240" height="220" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="3" stroke-dasharray="5,3" /> + <text x="890" y="220" font-family="Inter, sans-serif" font-size="16" text-anchor="middle" fill="#255200" font-weight="bold">Container-583</text> + <text x="890" y="240" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#255200">(Replicated across 3 DNs)</text> + + <!-- Blocks in Container 1 of Datanode 3 --> + <rect x="790" y="260" width="90" height="130" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="835" y="282" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-84</text> + <text x="835" y="302" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 235)</text> + + <!-- Chunks in Block-84 (DN 3) --> + <rect x="800" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="817" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-37</text> + + <rect x="800" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="817" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-92</text> + + <rect x="840" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="857" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-15</text> + + <rect x="840" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="857" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-64</text> + + <rect x="900" y="260" width="90" height="130" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="945" y="282" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-129</text> + <text x="945" y="302" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 517)</text> + + <!-- Chunks in Block-129 (DN 3) --> + <rect x="910" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="927" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-23</text> + + <rect x="910" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="927" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-51</text> + + <rect x="950" y="320" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="967" y="343" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-88</text> + + <rect x="950" y="360" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="967" y="378" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-47</text> + + <!-- Container 4 in Datanode 3 (Not Replicated) --> + <rect x="770" y="430" width="240" height="210" rx="8" ry="8" fill="#96d26e" stroke="#3a8100" stroke-width="2" /> + <text x="890" y="460" font-family="Inter, sans-serif" font-size="18" text-anchor="middle" fill="#255200" font-weight="bold">Container-912</text> + + <!-- Blocks in Container 4 of Datanode 3 --> + <rect x="790" y="480" width="90" height="140" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="835" y="505" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-637</text> + <text x="835" y="525" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 842)</text> + + <!-- Chunks in Block-637 --> + <rect x="800" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="817" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-26</text> + + <rect x="800" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="817" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-58</text> + + <rect x="840" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="857" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-39</text> + + <rect x="840" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="857" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-72</text> + + <rect x="900" y="480" width="90" height="140" rx="6" ry="6" fill="#64ab35" stroke="#306900" stroke-width="2" /> + <text x="945" y="505" font-family="Inter, sans-serif" font-size="14" text-anchor="middle" fill="#ffffff" font-weight="bold">Block-422</text> + <text x="945" y="525" font-family="Inter, sans-serif" font-size="12" text-anchor="middle" fill="#ffffff">(Object 705)</text> + + <!-- Chunks in Block-422 --> + <rect x="910" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="927" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-11</text> + + <rect x="910" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="927" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-43</text> + + <rect x="950" y="540" width="35" height="35" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="967" y="563" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-27</text> + + <rect x="950" y="580" width="35" height="25" rx="4" ry="4" fill="#459800" stroke="#255200" stroke-width="1.5" /> + <text x="967" y="598" font-family="Inter, sans-serif" font-size="10" text-anchor="middle" fill="white" font-weight="bold">Ch-81</text> + + <!-- Replication arrows with better styling --> + <line x1="330" y1="300" x2="429" y2="300" stroke="#3a8100" stroke-width="2.5" stroke-dasharray="8,3"/> + <polygon points="423,296 430,300 423,304" fill="#255200"/> + <polygon points="336,296 329,300 336,304" fill="#255200"/> + + <line x1="670" y1="300" x2="769" y2="300" stroke="#3a8100" stroke-width="2.5" stroke-dasharray="8,3"/> + <polygon points="763,296 770,300 763,304" fill="#255200"/> + <polygon points="676,296 669,300 676,304" fill="#255200"/> + +</svg> diff --git a/static/img/ozone/recon.svg b/static/img/ozone/recon.svg new file mode 100644 index 00000000..0bbcda8a --- /dev/null +++ b/static/img/ozone/recon.svg @@ -0,0 +1,90 @@ +<svg width="1200" height="550" xmlns="http://www.w3.org/2000/svg"; font-family="JetBrains Mono, monospace" font-size="12"> + <!-- Background --> + <rect width="1200" height="550" fill="#f7fcf0" /> + + <!-- Title --> + <text x="600" y="30" font-family="JetBrains Mono, monospace" font-size="22" text-anchor="middle" fill="#357500" font-weight="bold">Apache Ozone Recon Interactions</text> + + <defs> + <!-- Arrowhead marker definition --> + <marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"> + <path d="M 0 0 L 10 5 L 0 10 z" fill="#255200" /> + </marker> + </defs> + + <style> + .user-icon { stroke: #255200; stroke-width: 2; fill: none; } + .interface-box { fill: #d1e4bc; stroke: #3a8100; stroke-width: 2; rx: 8; ry: 8; } + .interface-text { fill: #255200; text-anchor: middle; font-weight: bold; } + /* Updated enclosing box style */ + .enclosing-box { fill: none; stroke: #2d6300; stroke-width: 1.5; stroke-dasharray: 6 4; rx: 10; ry: 10; } + .client-box { fill: #96d26e; stroke: #3a8100; stroke-width: 2.5; rx: 8; ry: 8; } + /* Updated client text style */ + .client-text { fill: #255200; text-anchor: middle; font-weight: bold; } + .arrow-line { stroke: #255200; stroke-width: 1.5; marker-end: url(#arrow); } + /* Ozone Manager style */ + .manager-box { fill: #4c8c2b; stroke: #306900; stroke-width: 2.5; rx: 8; ry: 8; } + .manager-text { fill: #ffffff; text-anchor: middle; font-weight: bold; } + /* SCM style */ + .scm-box { fill: #3a7a1a; stroke: #306900; stroke-width: 2.5; rx: 8; ry: 8; } + /* Datanode style */ + .datanode-box { fill: #5fa638; stroke: #3a8100; stroke-width: 2; rx: 8; ry: 8; } + /* Recon style */ + .recon-box { fill: #2a6010; stroke: #306900; stroke-width: 2.5; rx: 8; ry: 8; } + .recon-text { fill: #ffffff; text-anchor: middle; font-weight: bold; } + </style> + + <!-- Metadata Enclosing Box --> + <rect x="150" y="50" width="900" height="150" rx="15" ry="15" stroke="#357500" stroke-width="2" fill="#e0edd2"/> + <text x="600" y="70" font-family="JetBrains Mono, monospace" font-size="18" text-anchor="middle" fill="#357500" font-weight="bold">Metadata</text> + + <!-- Ozone Manager Box --> + <rect x="230" y="90" width="300" height="80" class="manager-box"/> + <text x="380" y="135" class="manager-text" font-size="16" font-family="JetBrains Mono, monospace">Ozone Manager</text> + + <!-- Storage Container Manager Box --> + <rect x="670" y="90" width="300" height="80" class="scm-box"/> + <text x="820" y="135" class="manager-text" font-size="16" font-family="JetBrains Mono, monospace">Storage Container Manager</text> + + <!-- Recon Box in the center --> + <rect x="450" y="230" width="300" height="80" class="recon-box"/> + <text x="600" y="260" class="recon-text" font-size="18" font-family="JetBrains Mono, monospace">Recon</text> + <text x="600" y="285" class="recon-text" font-size="14" font-family="JetBrains Mono, monospace">Analytics and Monitoring</text> + + <!-- Data Enclosing Box --> + <rect x="150" y="340" width="900" height="180" rx="15" ry="15" stroke="#357500" stroke-width="2" fill="#e0edd2"/> + <text x="600" y="360" font-family="JetBrains Mono, monospace" font-size="18" text-anchor="middle" fill="#357500" font-weight="bold">Data</text> + + <!-- Datanode Boxes: 3x2 Grid --> + <!-- Row 1 --> + <rect x="230" y="380" width="160" height="60" class="datanode-box"/> + <text x="310" y="415" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="520" y="380" width="160" height="60" class="datanode-box"/> + <text x="600" y="415" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="810" y="380" width="160" height="60" class="datanode-box"/> + <text x="890" y="415" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <!-- Row 2 --> + <rect x="230" y="450" width="160" height="60" class="datanode-box"/> + <text x="310" y="485" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="520" y="450" width="160" height="60" class="datanode-box"/> + <text x="600" y="485" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <rect x="810" y="450" width="160" height="60" class="datanode-box"/> + <text x="890" y="485" class="manager-text" font-size="14" font-family="JetBrains Mono, monospace">Datanode</text> + + <!-- Arrows from Recon to OM --> + <path d="M 500 230 L 380 170" stroke="#255200" stroke-width="2.5" stroke-dasharray="5,3" fill="none" marker-end="url(#arrow)"/> <!-- Recon -> OM --> + <text x="400" y="220" font-size="12" font-family="JetBrains Mono, monospace" fill="#255200" font-weight="bold">Read DB</text> + + <!-- Arrows from Recon to SCM --> + <path d="M 700 230 L 820 170" stroke="#255200" stroke-width="2.5" stroke-dasharray="5,3" fill="none" marker-end="url(#arrow)"/> <!-- Recon -> SCM --> + <text x="750" y="220" font-size="12" font-family="JetBrains Mono, monospace" fill="#255200" font-weight="bold">Read DB</text> + + <!-- Arrows from Data to Recon --> + <path d="M 600 340 L 600 310" stroke="#255200" stroke-width="2.5" stroke-dasharray="5,3" fill="none" marker-end="url(#arrow)"/> <!-- Data -> Recon --> + <text x="620" y="330" font-size="12" font-family="JetBrains Mono, monospace" fill="#255200" font-weight="bold">Heartbeats and Reports</text> +</svg> \ No newline at end of file --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
