errose28 commented on code in PR #516: URL: https://github.com/apache/ozone-site/pull/516#discussion_r3669061230
########## docs/05-administrator-guide/02-configuration/04-performance/12-short-circuit-local-reads.md: ########## @@ -0,0 +1,75 @@ +--- +sidebar_label: Short-Circuit Local Reads +--- + +# Short-Circuit Local Read + +By default, client reads data over gRPC from the Datanode. When the client asks the Datanode to read a file, the Datanode reads that file off of the disk and sends the data to the client over a gRPC connection. + +This short-circuit local read feature will bypass the Datanode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server. + +Short-circuit local read can provide a substantial performance boost to many applications by removing the overhead of network communication. + +## Prerequisite + +Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the Datanodes to communicate. + +The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details. + +The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the release package of the corresponding Hadoop version, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`. + +The `ozone checknative` command can be used to detect whether `libhadoop.so` can be found and loaded successfully by Ozone service. + +## Configuration + +Short-circuit local reads need to be configured on both the Datanode and the client. By default, it is disabled. + +```XML +<property> + <name>ozone.client.read.short-circuit</name> + <value>false</value> + <description>Disable or enable the short-circuit local read feature.</description> +</property> +``` + +It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket. + +```XML +<property> + <name>ozone.domain.socket.path</name> + <value></value> + <description> This is a path to a UNIX domain socket that will be used for + communication between the Datanode and local Ozone clients. + If the string "_PORT" is present in this path, it will be replaced by the TCP port of the Datanode. + </description> +</property> +``` + +The Datanode needs to be able to create this path. On the other hand, it should not be possible for any user except the user who launches Ozone service or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used. + +If you configure the `ozone.domain.socket.path` to some value, for example `/dir1/dir2/ozone_dn_socket`, please make sure that both `dir1` and `dir2` are existing directories, but the file `ozone_dn_socket` does not exist under `dir2`. `ozone_dn_socket` will be created by Ozone Datanode later during its startup. + +### Example Configuration + +To enable short-circuit read, here is an example configuration. + +```XML +<property> + <name>ozone.client.read.short-circuit</name> + <value>true</value> +</property> +<property> + <name>ozone.domain.socket.path</name> + <value>/var/run/ozone_dn_socket</value> +</property> +``` + +### Security Consideration + +To ensure data security and integrity, Ozone will follow the same rules as Hadoop to check permission on the `ozone.domain.socket.path` path as documented in [Socket Path Security](https://wiki.apache.org/hadoop/SocketPathSecurity). It will fail the `ozone.domain.socket.path` verification and disable the feature if the filesystem permissions of the specified path are inadequate. The verification failure message carries detail instruction about how to fix the problem. Following is an example: + +:::note +The path component: '/etc/Hadoop' in '/etc/Hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/Hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity Review Comment: These should be code blocks: ```suggestion The path component: `/etc/hadoop` in `/etc/hadoop/ozone_dn_socket` has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: `chmod o-w /etc/hadoop`. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity ``` ########## docs/08-developer-guide/04-project/04-enhancement-proposal.md: ########## @@ -0,0 +1,171 @@ +--- +sidebar_label: Ozone Enhancement Proposals +--- + +# Ozone Enhancement Proposals + +This document proposes a standard process for creating, reviewing, publishing, and maintaining design proposals for the Apache Ozone project. + +## Problem statement + +Some of the bigger features requires well defined plans before the implementation. Until now it was managed by uploading PDF design docs to selected Jira. There are multiple problems with the current practice. + +1. There is no easy way to find existing up-to-date and outdated design docs. +2. Design docs usually have better description of the problem that the user docs +3. We need better tools to discuss the design docs in the development phase of the doc + +We propose to follow the same process what we have now, but instead of uploading a PDF to the Jira, create a PR to merge the proposal document to the documentation project. + +## Non-goals + +- Modify the existing workflow or approval process +- Migrate existing documents +- Make it harder to create design docs (it should be easy to support the creation of proposals for any kind of tasks) +- Define how the design docs are handled/created *before* the publication (this proposal is about the publishing process) + +## Proposed solution + +- Open a dedicated Jira (`HDDS-*` but with specific component) +- Use standard name prefix in the Jira (easy to filter on the mailing list) `[OEP] +- Create a PR to add the design doc to the current documentation + - The content of the design can be added to the documentation (Recommended) + - Or can be added as external reference +- The design doc (or the summary with the reference) will be merged to the design doc folder of `hadoop-hdds/docs/content/design` (will be part of the docs) +- Discuss it as before (lazy consesus, except if somebody calls for a real vote) +- Design docs can be updated according to the changes during the implementation +- Only the implemented design docs will be visible as part of the design docs + +As a result all the design docs can be listed under the documentation page. + +A good design doc has the following properties: + +1. Publicly available for anybody (Please try to avoid services which are available only with registration, eg: google docs) +2. Archived for the future (Commit it to the source OR use Apache Jira or wiki) +3. Editable later (Best format is markdown, RTF is also good. PDF has a limitation, it's very hard to reuse the text, or create an updated design doc) +4. Well structured to make it easy to comment any part of the document (Markdown files which are part of the pull request can be commented in the PR line by line) + +### Example 1: Design doc as a markdown file + +The easiest way to create a design doc is to create a new markdown file in a PR and merge it to `hadoop-hdds/docs/content/design`. + +1. Publicly available: YES, it can be linked from Apache git or github +2. Archived: YES, and it's also versioned. All the change history can be tracked. +3. Editable later: YES, as it's just a simple text file +4. Commentable: YES, comment can be added to each line. + +### Example 2: Design doc as a PDF + +A very common practice of today is to create design doc on google docs and upload it to the Jira. + +1. Publicly available: YES, anybody can download it from the Jira. +2. Archived: YES, it's available from Apache infra. +3. Editable: NO, It's harder to reuse the text to import to the docs or create a new design doc. +4. Commentable: PARTIAL, Not as easy as a text file or the original google docs, but a good structure with numbered section may help + +### The format + +While the first version (markdown files) are the most powerful, the second version (the existing practice) is also acceptable. In this case we propose to create a PR with adding a reference page *without* the content but including the link. + +For example: + +```yaml +--- +title: Ozone Security Design +summary: A comprehensive description of the security flow between server and client components. +date: 2018-02-22 +jira: HDDS-4 +status: implemented +author: Sanjay Radia, Jitendra Pandey, Xiaoyu Yao, Anu Engineer + +## Summary + +Ozone security model is based on Kerberos and similar to the Hadoop security but some of the parts are improved: for example the SCM works as a Certificate Authority and PKI based solutions are wildely used. + +## Reference + +For more details please check the (uploaded design doc)[https://issues.apache.org/jira/secure/attachment/12911638/HadoopStorageLayerSecurity.pdf]. + +``` + +Obviously with the first approach the design doc itself can be included in this markdown file. + +## Migration + +It's not a hard requirement to migrate all the design doc. But process is always open: + +1. To create reference pages for any of the old design docs +2. To migrate any new design docs to markdown formats (by anybody not just by the author) +3. To update any of the old design docs based on the current state of the code (We have versioning!) + +## Document template + +This the proposed template to document any proposal. It's recommended but not required the use exactly the some structure. Some proposal may require different structure, but we need the following information. + +1. Summary + +> Give a one sentence summary, like the Jira title. It will be displayed on the documentation page. Should be enough to understand + +1. Status + +Defined in the markdown header. Proposed statuses: + +- `accepted`: (Use this as by default. If not accepted, won't be merged) +- `implemented`: The discussed technical solution is implemented (maybe with some minor implementation difference) +- `replaced`: Replaced by a new design doc +- `outdated`: Code has been changed and design doc doesn't reflect any more the state of the current code. + +Note: the *accepted* design docs won't be visible as part of the documentation or only under a dedicated section to clearly communicate that it's not ready, yet. + +1. Problem statement (Motivation / Abstract) Review Comment: I think the indentation under the numbers is wrong, which is why they aren't being recognized as a continuous list. ########## docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/16-hdds-8342-s3-lifecycle-configurations.md: ########## @@ -0,0 +1,77 @@ +# HDDS-8342 : S3 Lifecycle Configuration - Object Expiration + +Epic: [HDDS-8342](https://issues.apache.org/jira/browse/HDDS-8342) +Feature branch: https://github.com/apache/ozone/tree/HDDS-8342 + +## 1. Builds/intermittent test failures + +There are no intermittent failures specific to the HDDS-8342 branch as of now. During the development , it was ensured all the CI checks were clean prior to every commit merge . + +The plan is to run repeated CI checks on the merge commit to master. + +## 2. Documentation + +[User Documentation](https://ozone.apache.org/docs/next/administrator-guide/operations/object-lifecycle) for S3 Lifecycle Configuration - Object Expiration has been added. + +## 3. Design, attached the docs + +Design document can be found here : [S3 Lifecycle Configuration - Object Expiration Design Doc](https://github.com/apache/ozone/blob/master/hadoop-hdds/docs/content/design/s3-object-lifecycle-management.md). + +## 4. S3 compatibility + +N/A, only the object expiration action of S3 Lifecycle configuration is supported. + +## 5. Docker-compose / Acceptance tests + +New robot test [`om-lifecycle.robot`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/smoketest/lifecycle/om-lifecycle.robot) and [`bucketlifecycle.robot`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/smoketest/s3/bucketlifecycle.robot) are being added. + +Comprehensive tests with fault injection were added. The major one is [TestKeyLifecycleService](https://github.com/apache/ozone/blob/HDDS-8342/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyLifecycleService.java). + +## 6. Support of containers / Kubernetes + +No addition. No change in existing support. + +## 7. Coverage / Code quality + +[New Code Coverage](https://sonarcloud.io/summary/new_code?id=hadoop-ozone&branch=HDDS-8342) for S3 Lifecycle Configuration - Object Expiration (HDDS-8342) is 80.54% and [Overall Code Coverage](https://sonarcloud.io/summary/overall?id=hadoop-ozone&branch=HDDS-8342) is 78.4 . +[Overall Code Coverage](https://sonarcloud.io/summary/overall?id=hadoop-ozone&branch=master) for master is 78.2. + +## 8. Build time + +[Build time for the latest commit](https://github.com/ChenSammi/ozone/actions/runs/29250637888/job/86975578390) from HDDS-8342 Branch is 10m 12s. +[Build time for the latest commit](https://github.com/apache/ozone/actions/runs/29310061591/job/87011782215) from the master branch is 10m 21s. + +## 9. Possible incompatible changes/used feature flag + +A global enable/disable switch for this feature is added in HDDS-12780. [Implement the Lifecycle Service](https://issues.apache.org/jira/browse/HDDS-12780). + +To enable the feature, the following configs need to be added to OM Ozone-site.xml, and set the value to "true" Review Comment: ```suggestion To enable the feature, the following configs need to be added to OM `ozone-site.xml`, and set the value to `true` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
