cshuo commented on code in PR #20243: URL: https://github.com/apache/hudi/pull/20243#discussion_r4395989691
########## hudi-lakehouse/charts/hudi-spark-connect/README.md: ########## @@ -0,0 +1,361 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Hudi Spark Connect endpoint + +Deploys Spark Connect 4.1.3 with Hudi 1.2.0, Scala 2.13, and Java 17. +Tables use an external Hive Metastore and S3-compatible storage. +See the [Dockerfile](../../images/spark-connect/Dockerfile) for dependency versions. +The delivery scope is interactive SQL reads and writes, not general Spark job +submission or a required DataFrame-write integration. + +## Local-dev quickstart + +Requires Docker, minikube, Helm, and kubectl. Run from the repository root: + +```bash +# Start MinIO, Hive Metastore, and Spark Connect +./hudi-lakehouse/scripts/quickstart.sh --spark-connect + +# Also write example data and query it through Spark Connect (requires Maven and a JDK) +./hudi-lakehouse/scripts/quickstart.sh --run-example --spark-connect +``` + +The example uses the existing Spark writer to populate and register `default.trips`, +then prints city counts: `chennai = 33`, `san_francisco = 34`, `sao_paulo = 33`. +The query runs through PySpark inside the endpoint pod; no host client is needed. +Omit `--spark-connect` to query through Trino instead. + +To rebuild the image and reload the running endpoint: + +```bash +./hudi-lakehouse/scripts/quickstart.sh --spark-connect --rebuild-images +kubectl -n hudi-lakehouse rollout restart deployment/hudi-spark-connect +kubectl -n hudi-lakehouse rollout status deployment/hudi-spark-connect +``` + +## Install against existing infrastructure + +Build an image and supply your metastore, warehouse, and storage credentials: + +```bash +./hudi-lakehouse/scripts/build-connect-image.sh --registry my.registry/team --push +helm upgrade --install hudi-spark-connect hudi-lakehouse/charts/hudi-spark-connect \ + --namespace hudi-lakehouse --create-namespace \ + --set image.repository=my.registry/team/hudi-lakehouse-spark-connect \ + --set catalog.metastoreUri=thrift://my-hms:9083 \ + --set catalog.warehouse=s3a://my-bucket/connect \ + --set storage.s3.existingSecret=my-s3-credentials \ + --wait --timeout 10m +``` + +The Secret must be in the same namespace and contain `accessKey` and `secretKey`. +For temporary credentials, also set `storage.s3.sessionTokenKey`. +Without a Secret, Hadoop uses its default credential provider chain. +Local-dev credentials are for development only. + +Other settings in [values.yaml](values.yaml): + +| Setting | Purpose | +|---|---| +| `storage.s3.endpoint`, `region`, `pathStyleAccess` | MinIO or custom S3 endpoints | +| `spark.cores` | Local worker threads | +| `spark.master` | Empty means `local[cores]`; opt in to a remote master explicitly | +| `spark.driverMemory` | JVM heap; keep the pod memory limit higher | +| `resources` | Driver pod CPU, memory, and optional ephemeral-storage requests/limits | +| `scratch.sizeLimit` | Optional disk-backed `/tmp` volume limit, e.g. `20Gi`; empty by default | +| `spark.extraConf` | Additional Spark properties; no credentials or overrides of chart-owned settings | +| `hudi.defaults` | Server-side `hoodie.*` defaults loaded from `HUDI_CONF_DIR/hudi-defaults.conf` | +| `serviceAccount.name`, `automountToken` | Existing driver account and token mounting; required for Kubernetes executors | + +By default the endpoint runs one shared driver in local mode, without separate executors. +Keep one replica. Restarting it ends active sessions; tables remain in the +external metastore and object store. + +Size `/tmp` for peak concurrent shuffle/spill and temporary write data, with headroom. +For example, `--set scratch.sizeLimit=20Gi` caps the driver's scratch volume; it does +not reserve disk. Set `resources.requests.ephemeral-storage` for scheduling and +`resources.limits.ephemeral-storage` for the pod budget, allowing for logs and the +writable container layer as well. Node disk pressure can evict the pod before the +volume reaches its limit. Scratch data is lost with the pod; this is not table storage. +In cluster mode, size executor scratch separately; this setting applies only to the driver. + +The service has no authentication or TLS. Keep it on a trusted network: +any client that can connect can read and write using the server's storage credentials. +In Kubernetes executor mode, this trust also extends to the driver's ServiceAccount +permissions, including creating executor pods through the Kubernetes API. + +## Hudi writer defaults and table services + +The generic chart does not pin a table version. The local-dev values set: + +```yaml +hudi: + defaults: + hoodie.write.table.version: "8" + hoodie.write.auto.upgrade: "false" +``` + +This keeps new tables compatible with the local-dev Trino reader. Hudi 1.2.0 +otherwise defaults to table version 9. These are defaults, not enforcement: +explicit session/table settings can override them. Changing them does not downgrade +existing tables. Verify the versions supported by every reader before changing the pin. +The smoke test deliberately does not repeat the pin in `CREATE TABLE`. + +Use `hudi.defaults` for deployment-wide defaults, `TBLPROPERTIES` for table-specific +settings, and session `SET` for a writer operation. For example: + +```sql +CREATE TABLE orders (id INT, amount INT, ts BIGINT) USING hudi +TBLPROPERTIES (type = 'cow', primaryKey = 'id', preCombineField = 'ts', + 'hoodie.clustering.inline' = 'false'); + +SET hoodie.clean.automatic = true; +INSERT INTO orders VALUES (1, 100, 1); +RESET hoodie.clean.automatic; +``` + +Trino needs partition metadata in HMS as well as the Spark catalog entry. Ensure Review Comment: Thanks, Siva. Updated the README to clarify that Spark SQL writes enable Hive sync with the chart's default Hive catalog and use HMS mode by default. I also removed the explicit sync properties from the partitioned-table test. The live run passed: Trino read back both partitions with the expected rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
