dombizita commented on code in PR #203: URL: https://github.com/apache/ozone-site/pull/203#discussion_r2681693151
########## docs/04-user-guide/03-integrations/02-hue.md: ########## @@ -1,3 +1,235 @@ -# Hue +--- +sidebar_label: Hue +--- -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +# Integrating Apache Hue with Ozone + +Apache Hue provides a user-friendly web interface for interacting with various Hadoop ecosystem components, including file browsing. Hue can be configured to browse and manage data stored in Apache Ozone, leveraging Ozone's **HttpFS** interface, which offers WebHDFS-compatible REST endpoints. + +## How Hue Interacts with Storage + +Hue's File Browser and other components rely on Hadoop-compatible filesystem interfaces to: + +- Browse directory structures. +- List files and directories with their metadata. +- Upload and download files. +- Perform basic file management operations (create directory, rename, move, copy, delete). +- Provide paths for data access to integrated query engines (like Hive, Impala). + +## Ozone's HttpFS Interface for Hue + +Ozone enables Hue integration through its built-in **HttpFS service**, which typically runs as part of the Ozone Manager (OM). + +- **WebHDFS Compatibility:** The HttpFS service exposes a REST API at `/webhdfs/v1` that mimics the HDFS WebHDFS API. Hue uses this API to perform filesystem operations. +- **Translation:** HttpFS receives HTTP requests from Hue and translates them into Ozone RPC calls to the Ozone Manager. +- **Authentication:** Supports Kerberos (SPNEGO) for secure clusters, allowing Hue to authenticate securely. +- **Impersonation:** Supports Hadoop's proxy user mechanism, allowing the Hue service user to perform operations on behalf of the logged-in Hue user. + +:::info Note +While Hue might be configured with `ofs://` as its default filesystem (`fs_defaultfs`) for linking with query engines, the **File Browser** functionality primarily uses the **HttpFS/WebHDFS** endpoint (`webhdfs_url`) to interact with Ozone's namespace. +::: + +## Configuration Requirements + +### 1. Ozone HttpFS Configuration + +Ensure the Ozone Manager's HTTP/HTTPS interface is enabled and configured correctly in `ozone-site.xml`. HttpFS runs as part of the OM. + +```xml +<configuration> + + <!-- Ensure OM HTTP(S) address is configured --> + <property> + <name>ozone.om.http.address</name> + <value>om-host.example.com:9874</value> + <description>Ozone Manager HTTP address.</description> + </property> + <property> + <name>ozone.om.https.address</name> + <value>om-host.example.com:9875</value> + <description>Ozone Manager HTTPS address.</description> + </property> + <property> + <name>ozone.om.http.enabled</name> + <value>true</value> <!-- Or false if only using HTTPS --> + <description>Enable OM HTTP endpoint.</description> + </property> + <property> + <name>hdds.http.policy</name> + <value>HTTP_ONLY</value> <!-- Or HTTPS_ONLY, HTTP_AND_HTTPS --> + <description>Policy for HTTP/HTTPS endpoints.</description> + </property> + + <!-- Kerberos Authentication for HttpFS (if cluster is secure) --> + <property> + <name>ozone.om.http.auth.type</name> + <value>kerberos</value> + <description>Authentication type for OM HTTP endpoint.</description> + </property> + <property> + <name>ozone.om.http.kerberos.principal</name> + <value>HTTP/[email protected]</value> + <description>OM HTTP Kerberos principal (SPNEGO).</description> + </property> + <property> + <name>ozone.om.http.kerberos.keytab.file</name> + <value>/etc/security/keytabs/om-http.keytab</value> <!-- Path to OM HTTP keytab --> + <description>OM HTTP Kerberos keytab file.</description> + </property> + +</configuration> +``` + +- Adjust hostnames, ports, security settings, and keytab paths according to your cluster setup. +- Restart Ozone Manager after making changes. + +### 2. Hadoop Proxy User Configuration for Hue + +To allow the Hue service user (e.g., `hue`) to impersonate end-users when accessing Ozone via HttpFS, configure Hadoop's proxy user settings in the `core-site.xml` used by the Ozone Manager. + +```xml +<configuration> + + <property> + <name>hadoop.proxyuser.hue.hosts</name> + <!-- List of hosts where Hue service runs, or '*' for any host --> + <value>hue-host.example.com,*</value> + <description>Allow the 'hue' user to proxy requests from these hosts.</description> + </property> + + <property> + <name>hadoop.proxyuser.hue.groups</name> + <!-- List of groups whose members the 'hue' user can impersonate, or '*' for any group --> + <value>*</value> + <description>Allow the 'hue' user to impersonate users belonging to these groups.</description> + </property> + + <!-- Repeat for other proxy users if necessary --> + +</configuration> +``` + +- Replace `hue` with the actual OS user running the Hue service. +- Replace `hue-host.example.com`with the actual hostname(s) where Hue runs. Using`*` is less secure but often simpler for initial setup. +- Restart Ozone Manager after modifying `core-site.xml`. + Review Comment: We could link https://github.com/apache/ozone/pull/9596 here once it's merged, as that config could also be needed for Hue. ########## docs/04-user-guide/03-integrations/02-hue.md: ########## @@ -1,3 +1,235 @@ -# Hue +--- +sidebar_label: Hue +--- -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +# Integrating Apache Hue with Ozone + +Apache Hue provides a user-friendly web interface for interacting with various Hadoop ecosystem components, including file browsing. Hue can be configured to browse and manage data stored in Apache Ozone, leveraging Ozone's **HttpFS** interface, which offers WebHDFS-compatible REST endpoints. + +## How Hue Interacts with Storage + +Hue's File Browser and other components rely on Hadoop-compatible filesystem interfaces to: + +- Browse directory structures. +- List files and directories with their metadata. +- Upload and download files. +- Perform basic file management operations (create directory, rename, move, copy, delete). +- Provide paths for data access to integrated query engines (like Hive, Impala). + +## Ozone's HttpFS Interface for Hue + +Ozone enables Hue integration through its built-in **HttpFS service**, which typically runs as part of the Ozone Manager (OM). + +- **WebHDFS Compatibility:** The HttpFS service exposes a REST API at `/webhdfs/v1` that mimics the HDFS WebHDFS API. Hue uses this API to perform filesystem operations. +- **Translation:** HttpFS receives HTTP requests from Hue and translates them into Ozone RPC calls to the Ozone Manager. +- **Authentication:** Supports Kerberos (SPNEGO) for secure clusters, allowing Hue to authenticate securely. +- **Impersonation:** Supports Hadoop's proxy user mechanism, allowing the Hue service user to perform operations on behalf of the logged-in Hue user. + +:::info Note +While Hue might be configured with `ofs://` as its default filesystem (`fs_defaultfs`) for linking with query engines, the **File Browser** functionality primarily uses the **HttpFS/WebHDFS** endpoint (`webhdfs_url`) to interact with Ozone's namespace. +::: + +## Configuration Requirements + +### 1. Ozone HttpFS Configuration + +Ensure the Ozone Manager's HTTP/HTTPS interface is enabled and configured correctly in `ozone-site.xml`. HttpFS runs as part of the OM. + +```xml +<configuration> + + <!-- Ensure OM HTTP(S) address is configured --> + <property> + <name>ozone.om.http.address</name> + <value>om-host.example.com:9874</value> + <description>Ozone Manager HTTP address.</description> + </property> + <property> + <name>ozone.om.https.address</name> + <value>om-host.example.com:9875</value> + <description>Ozone Manager HTTPS address.</description> + </property> + <property> + <name>ozone.om.http.enabled</name> + <value>true</value> <!-- Or false if only using HTTPS --> + <description>Enable OM HTTP endpoint.</description> + </property> + <property> + <name>hdds.http.policy</name> + <value>HTTP_ONLY</value> <!-- Or HTTPS_ONLY, HTTP_AND_HTTPS --> + <description>Policy for HTTP/HTTPS endpoints.</description> + </property> + + <!-- Kerberos Authentication for HttpFS (if cluster is secure) --> + <property> + <name>ozone.om.http.auth.type</name> + <value>kerberos</value> + <description>Authentication type for OM HTTP endpoint.</description> + </property> + <property> + <name>ozone.om.http.kerberos.principal</name> + <value>HTTP/[email protected]</value> + <description>OM HTTP Kerberos principal (SPNEGO).</description> + </property> + <property> + <name>ozone.om.http.kerberos.keytab.file</name> + <value>/etc/security/keytabs/om-http.keytab</value> <!-- Path to OM HTTP keytab --> + <description>OM HTTP Kerberos keytab file.</description> + </property> + +</configuration> +``` + +- Adjust hostnames, ports, security settings, and keytab paths according to your cluster setup. +- Restart Ozone Manager after making changes. + +### 2. Hadoop Proxy User Configuration for Hue + +To allow the Hue service user (e.g., `hue`) to impersonate end-users when accessing Ozone via HttpFS, configure Hadoop's proxy user settings in the `core-site.xml` used by the Ozone Manager. + +```xml +<configuration> + + <property> + <name>hadoop.proxyuser.hue.hosts</name> + <!-- List of hosts where Hue service runs, or '*' for any host --> + <value>hue-host.example.com,*</value> + <description>Allow the 'hue' user to proxy requests from these hosts.</description> + </property> + + <property> + <name>hadoop.proxyuser.hue.groups</name> + <!-- List of groups whose members the 'hue' user can impersonate, or '*' for any group --> + <value>*</value> + <description>Allow the 'hue' user to impersonate users belonging to these groups.</description> + </property> + + <!-- Repeat for other proxy users if necessary --> + +</configuration> +``` + +- Replace `hue` with the actual OS user running the Hue service. +- Replace `hue-host.example.com`with the actual hostname(s) where Hue runs. Using`*` is less secure but often simpler for initial setup. +- Restart Ozone Manager after modifying `core-site.xml`. + +### 3. Hue Configuration (`hue.ini`) + +Configure Hue to use Ozone's HttpFS endpoint and optionally set the default filesystem path. Edit the `[desktop]`and`[[ozone]]`sections in`hue.ini`: + +```ini +[desktop] +# Define the default filesystem for Hue applications (e.g., Hive, Impala jobs) +# Use ofs:// with your OM Service ID for HA or OM address for non-HA +fs_defaultfs=ofs://ozonecluster/ + +# Secret key for session signing (ensure this is set securely) +secret_key=YourSecretKeyForHueSessionSigning + +[[ozone]] +# This section configures the Ozone filesystem interface in Hue + + # URL for the Ozone Manager's HttpFS (WebHDFS compatible) endpoint + # Use https:// if TLS is enabled for OM HTTP endpoint + webhdfs_url=http://om-host.example.com:9874/webhdfs/v1 + + # For secure clusters using Kerberos/SPNEGO for HttpFS: + # security_enabled=true + + # For secure clusters using TLS/SSL: + # Set to the path of the CA certificate bundle if using custom CAs, + # or set to false to disable server certificate verification (INSECURE!). + # ssl_cert_ca_verify=true + # [[ssl]] + # cacerts=/path/to/ca_bundle.pem + + # Set the default cluster name (optional, cosmetic) + # nice_name="My Ozone Cluster" + +``` + +- Replace `ofs://ozonecluster/`with your correct`ofs` path prefix (using your OM service ID). +- Replace `http://om-host.example.com:9874` with the actual HTTP(S) address of your Ozone Manager. +- Uncomment and configure `security_enabled`and`ssl_cert_ca_verify` as needed for secure clusters. Review Comment: nit ```suggestion - Uncomment and configure `security_enabled` and `ssl_cert_ca_verify` as needed for secure clusters. ``` ########## docs/04-user-guide/03-integrations/02-hue.md: ########## @@ -1,3 +1,235 @@ -# Hue +--- +sidebar_label: Hue +--- -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +# Integrating Apache Hue with Ozone + +Apache Hue provides a user-friendly web interface for interacting with various Hadoop ecosystem components, including file browsing. Hue can be configured to browse and manage data stored in Apache Ozone, leveraging Ozone's **HttpFS** interface, which offers WebHDFS-compatible REST endpoints. + +## How Hue Interacts with Storage + +Hue's File Browser and other components rely on Hadoop-compatible filesystem interfaces to: + +- Browse directory structures. +- List files and directories with their metadata. +- Upload and download files. +- Perform basic file management operations (create directory, rename, move, copy, delete). +- Provide paths for data access to integrated query engines (like Hive, Impala). + +## Ozone's HttpFS Interface for Hue + +Ozone enables Hue integration through its built-in **HttpFS service**, which typically runs as part of the Ozone Manager (OM). + +- **WebHDFS Compatibility:** The HttpFS service exposes a REST API at `/webhdfs/v1` that mimics the HDFS WebHDFS API. Hue uses this API to perform filesystem operations. +- **Translation:** HttpFS receives HTTP requests from Hue and translates them into Ozone RPC calls to the Ozone Manager. +- **Authentication:** Supports Kerberos (SPNEGO) for secure clusters, allowing Hue to authenticate securely. +- **Impersonation:** Supports Hadoop's proxy user mechanism, allowing the Hue service user to perform operations on behalf of the logged-in Hue user. + +:::info Note +While Hue might be configured with `ofs://` as its default filesystem (`fs_defaultfs`) for linking with query engines, the **File Browser** functionality primarily uses the **HttpFS/WebHDFS** endpoint (`webhdfs_url`) to interact with Ozone's namespace. +::: + +## Configuration Requirements + +### 1. Ozone HttpFS Configuration + +Ensure the Ozone Manager's HTTP/HTTPS interface is enabled and configured correctly in `ozone-site.xml`. HttpFS runs as part of the OM. + +```xml +<configuration> + + <!-- Ensure OM HTTP(S) address is configured --> + <property> + <name>ozone.om.http.address</name> + <value>om-host.example.com:9874</value> + <description>Ozone Manager HTTP address.</description> + </property> + <property> + <name>ozone.om.https.address</name> + <value>om-host.example.com:9875</value> + <description>Ozone Manager HTTPS address.</description> + </property> + <property> + <name>ozone.om.http.enabled</name> + <value>true</value> <!-- Or false if only using HTTPS --> + <description>Enable OM HTTP endpoint.</description> + </property> + <property> + <name>hdds.http.policy</name> + <value>HTTP_ONLY</value> <!-- Or HTTPS_ONLY, HTTP_AND_HTTPS --> + <description>Policy for HTTP/HTTPS endpoints.</description> + </property> + + <!-- Kerberos Authentication for HttpFS (if cluster is secure) --> + <property> + <name>ozone.om.http.auth.type</name> + <value>kerberos</value> + <description>Authentication type for OM HTTP endpoint.</description> + </property> + <property> + <name>ozone.om.http.kerberos.principal</name> + <value>HTTP/[email protected]</value> + <description>OM HTTP Kerberos principal (SPNEGO).</description> + </property> + <property> + <name>ozone.om.http.kerberos.keytab.file</name> + <value>/etc/security/keytabs/om-http.keytab</value> <!-- Path to OM HTTP keytab --> + <description>OM HTTP Kerberos keytab file.</description> + </property> + +</configuration> +``` + +- Adjust hostnames, ports, security settings, and keytab paths according to your cluster setup. +- Restart Ozone Manager after making changes. + +### 2. Hadoop Proxy User Configuration for Hue + +To allow the Hue service user (e.g., `hue`) to impersonate end-users when accessing Ozone via HttpFS, configure Hadoop's proxy user settings in the `core-site.xml` used by the Ozone Manager. + +```xml +<configuration> + + <property> + <name>hadoop.proxyuser.hue.hosts</name> + <!-- List of hosts where Hue service runs, or '*' for any host --> + <value>hue-host.example.com,*</value> + <description>Allow the 'hue' user to proxy requests from these hosts.</description> + </property> + + <property> + <name>hadoop.proxyuser.hue.groups</name> + <!-- List of groups whose members the 'hue' user can impersonate, or '*' for any group --> + <value>*</value> + <description>Allow the 'hue' user to impersonate users belonging to these groups.</description> + </property> + + <!-- Repeat for other proxy users if necessary --> + +</configuration> +``` + +- Replace `hue` with the actual OS user running the Hue service. +- Replace `hue-host.example.com`with the actual hostname(s) where Hue runs. Using`*` is less secure but often simpler for initial setup. +- Restart Ozone Manager after modifying `core-site.xml`. + +### 3. Hue Configuration (`hue.ini`) + +Configure Hue to use Ozone's HttpFS endpoint and optionally set the default filesystem path. Edit the `[desktop]`and`[[ozone]]`sections in`hue.ini`: + +```ini +[desktop] +# Define the default filesystem for Hue applications (e.g., Hive, Impala jobs) +# Use ofs:// with your OM Service ID for HA or OM address for non-HA +fs_defaultfs=ofs://ozonecluster/ Review Comment: I thought this should be added under the `[[ozone]]` section. Could you please verify? ########## docs/04-user-guide/03-integrations/02-hue.md: ########## @@ -1,3 +1,235 @@ -# Hue +--- +sidebar_label: Hue +--- -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +# Integrating Apache Hue with Ozone + +Apache Hue provides a user-friendly web interface for interacting with various Hadoop ecosystem components, including file browsing. Hue can be configured to browse and manage data stored in Apache Ozone, leveraging Ozone's **HttpFS** interface, which offers WebHDFS-compatible REST endpoints. + +## How Hue Interacts with Storage + +Hue's File Browser and other components rely on Hadoop-compatible filesystem interfaces to: + +- Browse directory structures. +- List files and directories with their metadata. +- Upload and download files. +- Perform basic file management operations (create directory, rename, move, copy, delete). +- Provide paths for data access to integrated query engines (like Hive, Impala). + +## Ozone's HttpFS Interface for Hue + +Ozone enables Hue integration through its built-in **HttpFS service**, which typically runs as part of the Ozone Manager (OM). + +- **WebHDFS Compatibility:** The HttpFS service exposes a REST API at `/webhdfs/v1` that mimics the HDFS WebHDFS API. Hue uses this API to perform filesystem operations. +- **Translation:** HttpFS receives HTTP requests from Hue and translates them into Ozone RPC calls to the Ozone Manager. +- **Authentication:** Supports Kerberos (SPNEGO) for secure clusters, allowing Hue to authenticate securely. +- **Impersonation:** Supports Hadoop's proxy user mechanism, allowing the Hue service user to perform operations on behalf of the logged-in Hue user. + +:::info Note +While Hue might be configured with `ofs://` as its default filesystem (`fs_defaultfs`) for linking with query engines, the **File Browser** functionality primarily uses the **HttpFS/WebHDFS** endpoint (`webhdfs_url`) to interact with Ozone's namespace. +::: + +## Configuration Requirements + +### 1. Ozone HttpFS Configuration + +Ensure the Ozone Manager's HTTP/HTTPS interface is enabled and configured correctly in `ozone-site.xml`. HttpFS runs as part of the OM. + +```xml +<configuration> + + <!-- Ensure OM HTTP(S) address is configured --> + <property> + <name>ozone.om.http.address</name> + <value>om-host.example.com:9874</value> + <description>Ozone Manager HTTP address.</description> + </property> + <property> + <name>ozone.om.https.address</name> + <value>om-host.example.com:9875</value> + <description>Ozone Manager HTTPS address.</description> + </property> + <property> + <name>ozone.om.http.enabled</name> + <value>true</value> <!-- Or false if only using HTTPS --> + <description>Enable OM HTTP endpoint.</description> + </property> + <property> + <name>hdds.http.policy</name> + <value>HTTP_ONLY</value> <!-- Or HTTPS_ONLY, HTTP_AND_HTTPS --> + <description>Policy for HTTP/HTTPS endpoints.</description> + </property> + + <!-- Kerberos Authentication for HttpFS (if cluster is secure) --> + <property> + <name>ozone.om.http.auth.type</name> + <value>kerberos</value> + <description>Authentication type for OM HTTP endpoint.</description> + </property> + <property> + <name>ozone.om.http.kerberos.principal</name> + <value>HTTP/[email protected]</value> + <description>OM HTTP Kerberos principal (SPNEGO).</description> + </property> + <property> + <name>ozone.om.http.kerberos.keytab.file</name> + <value>/etc/security/keytabs/om-http.keytab</value> <!-- Path to OM HTTP keytab --> + <description>OM HTTP Kerberos keytab file.</description> + </property> + +</configuration> +``` + +- Adjust hostnames, ports, security settings, and keytab paths according to your cluster setup. +- Restart Ozone Manager after making changes. + +### 2. Hadoop Proxy User Configuration for Hue + +To allow the Hue service user (e.g., `hue`) to impersonate end-users when accessing Ozone via HttpFS, configure Hadoop's proxy user settings in the `core-site.xml` used by the Ozone Manager. + +```xml +<configuration> + + <property> + <name>hadoop.proxyuser.hue.hosts</name> + <!-- List of hosts where Hue service runs, or '*' for any host --> + <value>hue-host.example.com,*</value> + <description>Allow the 'hue' user to proxy requests from these hosts.</description> + </property> + + <property> + <name>hadoop.proxyuser.hue.groups</name> + <!-- List of groups whose members the 'hue' user can impersonate, or '*' for any group --> + <value>*</value> + <description>Allow the 'hue' user to impersonate users belonging to these groups.</description> + </property> + + <!-- Repeat for other proxy users if necessary --> + +</configuration> +``` + +- Replace `hue` with the actual OS user running the Hue service. +- Replace `hue-host.example.com`with the actual hostname(s) where Hue runs. Using`*` is less secure but often simpler for initial setup. +- Restart Ozone Manager after modifying `core-site.xml`. + +### 3. Hue Configuration (`hue.ini`) + +Configure Hue to use Ozone's HttpFS endpoint and optionally set the default filesystem path. Edit the `[desktop]`and`[[ozone]]`sections in`hue.ini`: + +```ini +[desktop] +# Define the default filesystem for Hue applications (e.g., Hive, Impala jobs) +# Use ofs:// with your OM Service ID for HA or OM address for non-HA +fs_defaultfs=ofs://ozonecluster/ + +# Secret key for session signing (ensure this is set securely) +secret_key=YourSecretKeyForHueSessionSigning Review Comment: Not sure about this config, do you know why is it needed? ########## docs/04-user-guide/03-integrations/02-hue.md: ########## @@ -1,3 +1,235 @@ -# Hue +--- +sidebar_label: Hue +--- -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +# Integrating Apache Hue with Ozone + +Apache Hue provides a user-friendly web interface for interacting with various Hadoop ecosystem components, including file browsing. Hue can be configured to browse and manage data stored in Apache Ozone, leveraging Ozone's **HttpFS** interface, which offers WebHDFS-compatible REST endpoints. + +## How Hue Interacts with Storage + +Hue's File Browser and other components rely on Hadoop-compatible filesystem interfaces to: + +- Browse directory structures. +- List files and directories with their metadata. +- Upload and download files. +- Perform basic file management operations (create directory, rename, move, copy, delete). +- Provide paths for data access to integrated query engines (like Hive, Impala). + +## Ozone's HttpFS Interface for Hue + +Ozone enables Hue integration through its built-in **HttpFS service**, which typically runs as part of the Ozone Manager (OM). + +- **WebHDFS Compatibility:** The HttpFS service exposes a REST API at `/webhdfs/v1` that mimics the HDFS WebHDFS API. Hue uses this API to perform filesystem operations. +- **Translation:** HttpFS receives HTTP requests from Hue and translates them into Ozone RPC calls to the Ozone Manager. +- **Authentication:** Supports Kerberos (SPNEGO) for secure clusters, allowing Hue to authenticate securely. +- **Impersonation:** Supports Hadoop's proxy user mechanism, allowing the Hue service user to perform operations on behalf of the logged-in Hue user. + +:::info Note +While Hue might be configured with `ofs://` as its default filesystem (`fs_defaultfs`) for linking with query engines, the **File Browser** functionality primarily uses the **HttpFS/WebHDFS** endpoint (`webhdfs_url`) to interact with Ozone's namespace. +::: + +## Configuration Requirements + +### 1. Ozone HttpFS Configuration + +Ensure the Ozone Manager's HTTP/HTTPS interface is enabled and configured correctly in `ozone-site.xml`. HttpFS runs as part of the OM. + +```xml +<configuration> + + <!-- Ensure OM HTTP(S) address is configured --> + <property> + <name>ozone.om.http.address</name> + <value>om-host.example.com:9874</value> + <description>Ozone Manager HTTP address.</description> + </property> + <property> + <name>ozone.om.https.address</name> + <value>om-host.example.com:9875</value> + <description>Ozone Manager HTTPS address.</description> + </property> + <property> + <name>ozone.om.http.enabled</name> + <value>true</value> <!-- Or false if only using HTTPS --> + <description>Enable OM HTTP endpoint.</description> + </property> + <property> + <name>hdds.http.policy</name> + <value>HTTP_ONLY</value> <!-- Or HTTPS_ONLY, HTTP_AND_HTTPS --> + <description>Policy for HTTP/HTTPS endpoints.</description> + </property> + + <!-- Kerberos Authentication for HttpFS (if cluster is secure) --> + <property> + <name>ozone.om.http.auth.type</name> + <value>kerberos</value> + <description>Authentication type for OM HTTP endpoint.</description> + </property> + <property> + <name>ozone.om.http.kerberos.principal</name> + <value>HTTP/[email protected]</value> + <description>OM HTTP Kerberos principal (SPNEGO).</description> + </property> + <property> + <name>ozone.om.http.kerberos.keytab.file</name> + <value>/etc/security/keytabs/om-http.keytab</value> <!-- Path to OM HTTP keytab --> + <description>OM HTTP Kerberos keytab file.</description> + </property> + +</configuration> +``` + +- Adjust hostnames, ports, security settings, and keytab paths according to your cluster setup. +- Restart Ozone Manager after making changes. + +### 2. Hadoop Proxy User Configuration for Hue + +To allow the Hue service user (e.g., `hue`) to impersonate end-users when accessing Ozone via HttpFS, configure Hadoop's proxy user settings in the `core-site.xml` used by the Ozone Manager. + +```xml +<configuration> + + <property> + <name>hadoop.proxyuser.hue.hosts</name> + <!-- List of hosts where Hue service runs, or '*' for any host --> + <value>hue-host.example.com,*</value> + <description>Allow the 'hue' user to proxy requests from these hosts.</description> + </property> + + <property> + <name>hadoop.proxyuser.hue.groups</name> + <!-- List of groups whose members the 'hue' user can impersonate, or '*' for any group --> + <value>*</value> + <description>Allow the 'hue' user to impersonate users belonging to these groups.</description> + </property> + + <!-- Repeat for other proxy users if necessary --> + +</configuration> +``` + +- Replace `hue` with the actual OS user running the Hue service. +- Replace `hue-host.example.com`with the actual hostname(s) where Hue runs. Using`*` is less secure but often simpler for initial setup. +- Restart Ozone Manager after modifying `core-site.xml`. + +### 3. Hue Configuration (`hue.ini`) + +Configure Hue to use Ozone's HttpFS endpoint and optionally set the default filesystem path. Edit the `[desktop]`and`[[ozone]]`sections in`hue.ini`: + +```ini +[desktop] +# Define the default filesystem for Hue applications (e.g., Hive, Impala jobs) +# Use ofs:// with your OM Service ID for HA or OM address for non-HA +fs_defaultfs=ofs://ozonecluster/ + +# Secret key for session signing (ensure this is set securely) +secret_key=YourSecretKeyForHueSessionSigning + +[[ozone]] +# This section configures the Ozone filesystem interface in Hue + + # URL for the Ozone Manager's HttpFS (WebHDFS compatible) endpoint + # Use https:// if TLS is enabled for OM HTTP endpoint + webhdfs_url=http://om-host.example.com:9874/webhdfs/v1 Review Comment: This looks incorrect, the webhdfs_url should be the HttpFS gateway endpoint. ```suggestion webhdfs_url=http(s)://[***OZONE-HTTPFS-HOST***]:[***OZONE-HTTPFS-PORT***]/webhdfs/v1 ``` ########## docs/04-user-guide/03-integrations/02-hue.md: ########## @@ -1,3 +1,235 @@ -# Hue +--- +sidebar_label: Hue +--- -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +# Integrating Apache Hue with Ozone + +Apache Hue provides a user-friendly web interface for interacting with various Hadoop ecosystem components, including file browsing. Hue can be configured to browse and manage data stored in Apache Ozone, leveraging Ozone's **HttpFS** interface, which offers WebHDFS-compatible REST endpoints. + +## How Hue Interacts with Storage + +Hue's File Browser and other components rely on Hadoop-compatible filesystem interfaces to: + +- Browse directory structures. +- List files and directories with their metadata. +- Upload and download files. +- Perform basic file management operations (create directory, rename, move, copy, delete). +- Provide paths for data access to integrated query engines (like Hive, Impala). + +## Ozone's HttpFS Interface for Hue + +Ozone enables Hue integration through its built-in **HttpFS service**, which typically runs as part of the Ozone Manager (OM). + +- **WebHDFS Compatibility:** The HttpFS service exposes a REST API at `/webhdfs/v1` that mimics the HDFS WebHDFS API. Hue uses this API to perform filesystem operations. +- **Translation:** HttpFS receives HTTP requests from Hue and translates them into Ozone RPC calls to the Ozone Manager. +- **Authentication:** Supports Kerberos (SPNEGO) for secure clusters, allowing Hue to authenticate securely. +- **Impersonation:** Supports Hadoop's proxy user mechanism, allowing the Hue service user to perform operations on behalf of the logged-in Hue user. + +:::info Note +While Hue might be configured with `ofs://` as its default filesystem (`fs_defaultfs`) for linking with query engines, the **File Browser** functionality primarily uses the **HttpFS/WebHDFS** endpoint (`webhdfs_url`) to interact with Ozone's namespace. +::: + +## Configuration Requirements + +### 1. Ozone HttpFS Configuration + +Ensure the Ozone Manager's HTTP/HTTPS interface is enabled and configured correctly in `ozone-site.xml`. HttpFS runs as part of the OM. + +```xml +<configuration> + + <!-- Ensure OM HTTP(S) address is configured --> + <property> + <name>ozone.om.http.address</name> + <value>om-host.example.com:9874</value> + <description>Ozone Manager HTTP address.</description> + </property> + <property> + <name>ozone.om.https.address</name> + <value>om-host.example.com:9875</value> + <description>Ozone Manager HTTPS address.</description> + </property> + <property> + <name>ozone.om.http.enabled</name> + <value>true</value> <!-- Or false if only using HTTPS --> + <description>Enable OM HTTP endpoint.</description> + </property> + <property> + <name>hdds.http.policy</name> + <value>HTTP_ONLY</value> <!-- Or HTTPS_ONLY, HTTP_AND_HTTPS --> + <description>Policy for HTTP/HTTPS endpoints.</description> + </property> + + <!-- Kerberos Authentication for HttpFS (if cluster is secure) --> + <property> + <name>ozone.om.http.auth.type</name> + <value>kerberos</value> + <description>Authentication type for OM HTTP endpoint.</description> + </property> + <property> + <name>ozone.om.http.kerberos.principal</name> + <value>HTTP/[email protected]</value> + <description>OM HTTP Kerberos principal (SPNEGO).</description> + </property> + <property> + <name>ozone.om.http.kerberos.keytab.file</name> + <value>/etc/security/keytabs/om-http.keytab</value> <!-- Path to OM HTTP keytab --> + <description>OM HTTP Kerberos keytab file.</description> + </property> + +</configuration> +``` + +- Adjust hostnames, ports, security settings, and keytab paths according to your cluster setup. +- Restart Ozone Manager after making changes. + +### 2. Hadoop Proxy User Configuration for Hue + +To allow the Hue service user (e.g., `hue`) to impersonate end-users when accessing Ozone via HttpFS, configure Hadoop's proxy user settings in the `core-site.xml` used by the Ozone Manager. + +```xml +<configuration> + + <property> + <name>hadoop.proxyuser.hue.hosts</name> + <!-- List of hosts where Hue service runs, or '*' for any host --> + <value>hue-host.example.com,*</value> + <description>Allow the 'hue' user to proxy requests from these hosts.</description> + </property> + + <property> + <name>hadoop.proxyuser.hue.groups</name> + <!-- List of groups whose members the 'hue' user can impersonate, or '*' for any group --> + <value>*</value> + <description>Allow the 'hue' user to impersonate users belonging to these groups.</description> + </property> + + <!-- Repeat for other proxy users if necessary --> + +</configuration> +``` + +- Replace `hue` with the actual OS user running the Hue service. +- Replace `hue-host.example.com`with the actual hostname(s) where Hue runs. Using`*` is less secure but often simpler for initial setup. +- Restart Ozone Manager after modifying `core-site.xml`. + +### 3. Hue Configuration (`hue.ini`) + +Configure Hue to use Ozone's HttpFS endpoint and optionally set the default filesystem path. Edit the `[desktop]`and`[[ozone]]`sections in`hue.ini`: + +```ini +[desktop] +# Define the default filesystem for Hue applications (e.g., Hive, Impala jobs) +# Use ofs:// with your OM Service ID for HA or OM address for non-HA +fs_defaultfs=ofs://ozonecluster/ + +# Secret key for session signing (ensure this is set securely) +secret_key=YourSecretKeyForHueSessionSigning + +[[ozone]] +# This section configures the Ozone filesystem interface in Hue + + # URL for the Ozone Manager's HttpFS (WebHDFS compatible) endpoint + # Use https:// if TLS is enabled for OM HTTP endpoint + webhdfs_url=http://om-host.example.com:9874/webhdfs/v1 + + # For secure clusters using Kerberos/SPNEGO for HttpFS: + # security_enabled=true + + # For secure clusters using TLS/SSL: + # Set to the path of the CA certificate bundle if using custom CAs, + # or set to false to disable server certificate verification (INSECURE!). + # ssl_cert_ca_verify=true + # [[ssl]] + # cacerts=/path/to/ca_bundle.pem + + # Set the default cluster name (optional, cosmetic) + # nice_name="My Ozone Cluster" + +``` + +- Replace `ofs://ozonecluster/`with your correct`ofs` path prefix (using your OM service ID). Review Comment: nit ```suggestion - Replace `ofs://ozonecluster/` with your correct `ofs` path prefix (using your OM service ID). ``` ########## docs/04-user-guide/03-integrations/02-hue.md: ########## @@ -1,3 +1,235 @@ -# Hue +--- +sidebar_label: Hue +--- -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +# Integrating Apache Hue with Ozone + +Apache Hue provides a user-friendly web interface for interacting with various Hadoop ecosystem components, including file browsing. Hue can be configured to browse and manage data stored in Apache Ozone, leveraging Ozone's **HttpFS** interface, which offers WebHDFS-compatible REST endpoints. + +## How Hue Interacts with Storage + +Hue's File Browser and other components rely on Hadoop-compatible filesystem interfaces to: + +- Browse directory structures. +- List files and directories with their metadata. +- Upload and download files. +- Perform basic file management operations (create directory, rename, move, copy, delete). +- Provide paths for data access to integrated query engines (like Hive, Impala). + +## Ozone's HttpFS Interface for Hue + +Ozone enables Hue integration through its built-in **HttpFS service**, which typically runs as part of the Ozone Manager (OM). + +- **WebHDFS Compatibility:** The HttpFS service exposes a REST API at `/webhdfs/v1` that mimics the HDFS WebHDFS API. Hue uses this API to perform filesystem operations. +- **Translation:** HttpFS receives HTTP requests from Hue and translates them into Ozone RPC calls to the Ozone Manager. +- **Authentication:** Supports Kerberos (SPNEGO) for secure clusters, allowing Hue to authenticate securely. +- **Impersonation:** Supports Hadoop's proxy user mechanism, allowing the Hue service user to perform operations on behalf of the logged-in Hue user. + +:::info Note +While Hue might be configured with `ofs://` as its default filesystem (`fs_defaultfs`) for linking with query engines, the **File Browser** functionality primarily uses the **HttpFS/WebHDFS** endpoint (`webhdfs_url`) to interact with Ozone's namespace. +::: + +## Configuration Requirements + +### 1. Ozone HttpFS Configuration + +Ensure the Ozone Manager's HTTP/HTTPS interface is enabled and configured correctly in `ozone-site.xml`. HttpFS runs as part of the OM. + +```xml +<configuration> + + <!-- Ensure OM HTTP(S) address is configured --> + <property> + <name>ozone.om.http.address</name> + <value>om-host.example.com:9874</value> + <description>Ozone Manager HTTP address.</description> + </property> + <property> + <name>ozone.om.https.address</name> + <value>om-host.example.com:9875</value> + <description>Ozone Manager HTTPS address.</description> + </property> + <property> + <name>ozone.om.http.enabled</name> + <value>true</value> <!-- Or false if only using HTTPS --> + <description>Enable OM HTTP endpoint.</description> + </property> + <property> + <name>hdds.http.policy</name> + <value>HTTP_ONLY</value> <!-- Or HTTPS_ONLY, HTTP_AND_HTTPS --> + <description>Policy for HTTP/HTTPS endpoints.</description> + </property> + + <!-- Kerberos Authentication for HttpFS (if cluster is secure) --> + <property> + <name>ozone.om.http.auth.type</name> + <value>kerberos</value> + <description>Authentication type for OM HTTP endpoint.</description> + </property> + <property> + <name>ozone.om.http.kerberos.principal</name> + <value>HTTP/[email protected]</value> + <description>OM HTTP Kerberos principal (SPNEGO).</description> + </property> + <property> + <name>ozone.om.http.kerberos.keytab.file</name> + <value>/etc/security/keytabs/om-http.keytab</value> <!-- Path to OM HTTP keytab --> + <description>OM HTTP Kerberos keytab file.</description> + </property> + +</configuration> +``` + +- Adjust hostnames, ports, security settings, and keytab paths according to your cluster setup. +- Restart Ozone Manager after making changes. + +### 2. Hadoop Proxy User Configuration for Hue + +To allow the Hue service user (e.g., `hue`) to impersonate end-users when accessing Ozone via HttpFS, configure Hadoop's proxy user settings in the `core-site.xml` used by the Ozone Manager. + +```xml +<configuration> + + <property> + <name>hadoop.proxyuser.hue.hosts</name> + <!-- List of hosts where Hue service runs, or '*' for any host --> + <value>hue-host.example.com,*</value> + <description>Allow the 'hue' user to proxy requests from these hosts.</description> + </property> + + <property> + <name>hadoop.proxyuser.hue.groups</name> + <!-- List of groups whose members the 'hue' user can impersonate, or '*' for any group --> + <value>*</value> + <description>Allow the 'hue' user to impersonate users belonging to these groups.</description> + </property> + + <!-- Repeat for other proxy users if necessary --> + +</configuration> +``` + +- Replace `hue` with the actual OS user running the Hue service. +- Replace `hue-host.example.com`with the actual hostname(s) where Hue runs. Using`*` is less secure but often simpler for initial setup. +- Restart Ozone Manager after modifying `core-site.xml`. + +### 3. Hue Configuration (`hue.ini`) + +Configure Hue to use Ozone's HttpFS endpoint and optionally set the default filesystem path. Edit the `[desktop]`and`[[ozone]]`sections in`hue.ini`: + +```ini +[desktop] +# Define the default filesystem for Hue applications (e.g., Hive, Impala jobs) +# Use ofs:// with your OM Service ID for HA or OM address for non-HA +fs_defaultfs=ofs://ozonecluster/ + +# Secret key for session signing (ensure this is set securely) +secret_key=YourSecretKeyForHueSessionSigning + +[[ozone]] +# This section configures the Ozone filesystem interface in Hue + + # URL for the Ozone Manager's HttpFS (WebHDFS compatible) endpoint + # Use https:// if TLS is enabled for OM HTTP endpoint + webhdfs_url=http://om-host.example.com:9874/webhdfs/v1 + + # For secure clusters using Kerberos/SPNEGO for HttpFS: + # security_enabled=true + + # For secure clusters using TLS/SSL: + # Set to the path of the CA certificate bundle if using custom CAs, + # or set to false to disable server certificate verification (INSECURE!). + # ssl_cert_ca_verify=true + # [[ssl]] + # cacerts=/path/to/ca_bundle.pem + + # Set the default cluster name (optional, cosmetic) + # nice_name="My Ozone Cluster" + +``` + +- Replace `ofs://ozonecluster/`with your correct`ofs` path prefix (using your OM service ID). +- Replace `http://om-host.example.com:9874` with the actual HTTP(S) address of your Ozone Manager. +- Uncomment and configure `security_enabled`and`ssl_cert_ca_verify` as needed for secure clusters. +- Restart the Hue service after modifying `hue.ini`. + +## Using Hue with Ozone via HttpFS (Recommended for Browsing) + +After successful configuration using HttpFS, users logging into Hue should be able to use the **File Browser** application to navigate the Ozone namespace with filesystem semantics. + +- **Browsing:** Navigate through volumes, buckets, and directories (especially in FSO buckets). +- **Operations:** Upload, download, create directories, rename, move, copy, delete files/directories (subject to user permissions in Ozone and limitations based on bucket layout). +- **File Viewing/Editing:** View and edit text-based files directly. + +Data stored in Ozone can also be accessed by other Hue applications like the **Hive** and **Impala** query editors by referencing tables whose `LOCATION`points to`ofs://`paths (configured via`fs_defaultfs` or explicitly in table definitions). + +## Using Hue with Ozone via S3 API (Alternative) + +Hue also supports browsing S3-compatible storage directly. You can configure Hue to connect to Ozone's S3 Gateway endpoint. This method is primarily useful for browsing **OBS (Object Store)** buckets or when S3 access patterns are preferred. Review Comment: Not sure about this whole section, I'm not aware that we have ever tested this. Do you know where this came from? Any resource mentioning this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
