Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
deniskuzZ commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2470008902 ## packaging/src/docker/thirdparties/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup + +This repository contains a docker-compose-based setup integrating Apache Hive, Gravitino Iceberg REST server, and Keycloak for OAuth2 authentication. It allows Hive to use an Iceberg REST catalog secured via Keycloak. + +## Table of Contents +- Architecture Overview +- Prerequisites +- Quickstart +- Configuration + - Keycloak + - Gravitino + - Hive +- Networking Notes + +## Architecture Overview +This diagram illustrates the key docker-compose components and their interactions in this setup: + +``` + oAuth2 (REST API) + +---+ + | | + | v +++--+ +---+ +-+ +| | RESTCatalog | | oauth2 | | +| Hive | (REST API) | Gravitino| (REST API) | Keycloak | +| (HiveServer2)+-->|Iceberg REST +--->| OAuth2 Auth| +| | | Server || Server | +++--+ +-+-+ +-+ + || + data | metadata files| + files ++ + | + v ++---+ +---+ +| | creates dir | | +| /warehouse|<--+ init| +| (Docker volume) | sets | container | +| | permissions | | ++---+ +---+ +``` + +- Hive: +- Runs HiveServer2, connects to Gravitino via Iceberg REST catalog. +- Write Iceberg data files to the shared warehouse volume. +- Gravitino: +- Exposes REST API for Iceberg catalog. +- Writes Iceberg metadata files to shared warehouse volume (.metadata.json). +- Doesn't supports serving as oauth2 provider, so this example uses an external OAuth2 provider (Keyclock). +- Keycloak: + - OAuth2 server providing authentication and token issuance for Hive/Gravitino. +- /warehouse: +- Shared Docker volume for Iceberg table data and metadata. +- Init container: +- Creates shared /warehouse folder and sets filesystem permissions as a one time initialization step. + +## Prerequisites +- Docker & Docker Compose +- Java (for local Hive beeline client) +- ```$HIVE_HOME``` environment variable pointing to Hive installation (for connecting to Beeline) + +## Quickstart +### STEP 1: Build Hive Docker Image +- Currently, only Hive 4.2.0-SNAPSHOT has the required Iceberg REST catalog client. +- To build the Hive Docker image with Iceberg REST catalog client on your local machine, follow these steps: +- Build Hive using Docker profile. +- Copy ```apache-hive-4.2.0-SNAPSHOT-bin.tar.gz``` to ```packaging/cache``` folder. Review Comment: why do we need to copy the jar? we are not doing that manually in the root hive docker-compose. docker profile is creating the hive images that we use in docker-compose -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
deniskuzZ commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2470008902 ## packaging/src/docker/thirdparties/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup + +This repository contains a docker-compose-based setup integrating Apache Hive, Gravitino Iceberg REST server, and Keycloak for OAuth2 authentication. It allows Hive to use an Iceberg REST catalog secured via Keycloak. + +## Table of Contents +- Architecture Overview +- Prerequisites +- Quickstart +- Configuration + - Keycloak + - Gravitino + - Hive +- Networking Notes + +## Architecture Overview +This diagram illustrates the key docker-compose components and their interactions in this setup: + +``` + oAuth2 (REST API) + +---+ + | | + | v +++--+ +---+ +-+ +| | RESTCatalog | | oauth2 | | +| Hive | (REST API) | Gravitino| (REST API) | Keycloak | +| (HiveServer2)+-->|Iceberg REST +--->| OAuth2 Auth| +| | | Server || Server | +++--+ +-+-+ +-+ + || + data | metadata files| + files ++ + | + v ++---+ +---+ +| | creates dir | | +| /warehouse|<--+ init| +| (Docker volume) | sets | container | +| | permissions | | ++---+ +---+ +``` + +- Hive: +- Runs HiveServer2, connects to Gravitino via Iceberg REST catalog. +- Write Iceberg data files to the shared warehouse volume. +- Gravitino: +- Exposes REST API for Iceberg catalog. +- Writes Iceberg metadata files to shared warehouse volume (.metadata.json). +- Doesn't supports serving as oauth2 provider, so this example uses an external OAuth2 provider (Keyclock). +- Keycloak: + - OAuth2 server providing authentication and token issuance for Hive/Gravitino. +- /warehouse: +- Shared Docker volume for Iceberg table data and metadata. +- Init container: +- Creates shared /warehouse folder and sets filesystem permissions as a one time initialization step. + +## Prerequisites +- Docker & Docker Compose +- Java (for local Hive beeline client) +- ```$HIVE_HOME``` environment variable pointing to Hive installation (for connecting to Beeline) + +## Quickstart +### STEP 1: Build Hive Docker Image +- Currently, only Hive 4.2.0-SNAPSHOT has the required Iceberg REST catalog client. +- To build the Hive Docker image with Iceberg REST catalog client on your local machine, follow these steps: +- Build Hive using Docker profile. +- Copy ```apache-hive-4.2.0-SNAPSHOT-bin.tar.gz``` to ```packaging/cache``` folder. Review Comment: why do we need to copy the jar? we are not doing that manually for the base hive docker-compose -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
deniskuzZ commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2469987834 ## packaging/src/docker/thirdparties/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup + +This repository contains a docker-compose-based setup integrating Apache Hive, Gravitino Iceberg REST server, and Keycloak for OAuth2 authentication. It allows Hive to use an Iceberg REST catalog secured via Keycloak. Review Comment: this package? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
difin commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2469930104 ## packaging/src/docker/thirdparties/polaris/README.md: ## @@ -0,0 +1,190 @@ + + +# Hive + Polaris: Docker-Compose Setup Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
difin commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2469794938 ## packaging/src/docker/examples/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup Review Comment: Hi @deniskuzZ, I created a PR for adding the quickstarts to hive-site: https://github.com/apache/hive-site/pull/68 Can you please review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
difin commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2467507278 ## packaging/src/docker/examples/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup Review Comment: I added that because the existing Hive docker-compose setup contains a README file: https://github.com/apache/hive/blob/master/packaging/src/docker/README.md So, just wanted to confirm: should we have these README files here for consistency, or only to add to the hive-site? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
difin commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2467507278 ## packaging/src/docker/examples/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup Review Comment: I added that because the existing Hive docker-compose setup contains a README file: https://github.com/apache/hive/blob/master/packaging/src/docker/README.md So, just wanted to confirm: do we want to have these README files on hive-site only or here also? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
deniskuzZ commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2462654063 ## packaging/src/docker/examples/polaris/README.md: ## @@ -0,0 +1,190 @@ + + +# Hive + Polaris: Docker-Compose Setup Review Comment: same -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
deniskuzZ commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2462653609 ## packaging/src/docker/examples/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup Review Comment: i think that should be part of Hive dev setup: https://hive.apache.org/development/quickstart/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
deniskuzZ commented on code in PR #6147: URL: https://github.com/apache/hive/pull/6147#discussion_r2462653609 ## packaging/src/docker/examples/gravitino/README.md: ## @@ -0,0 +1,248 @@ + + +# Hive + Gravitino + Keycloak: Docker-Compose Setup Review Comment: i think that should be part of Hive dev setup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HIVE-29285: Iceberg: Add docker-compose examples for REST Catalog int… [hive]
sonarqubecloud[bot] commented on PR #6147: URL: https://github.com/apache/hive/pull/6147#issuecomment-3434140940 ## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6147) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6147&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6147&issueStatuses=ACCEPTED) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6147&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6147&metric=new_coverage&view=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6147&metric=new_duplicated_lines_density&view=list) [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6147) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
