wombatu-kun commented on code in PR #20621:
URL: https://github.com/apache/hudi/pull/20621#discussion_r4573883714


##########
hudi-lakehouse/docker/compose.yaml:
##########
@@ -0,0 +1,167 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: hudi-lakehouse
+
+# Six default services and one optional Ollama service. Startup order:
+# minio -> minio-init -> hive-metastore -> spark-connect -> seed -> gateway.
+# minio-init and seed exit after completing their work; the other services 
stay running.
+services:
+  # S3-compatible storage for Hudi data, log files and timelines.
+  # S3 is internal at http://minio:9000; the console defaults to 
localhost:9001.
+  minio:
+    image: 
${MINIO_IMAGE:-cgr.dev/chainguard/minio:latest@sha256:fb893e36fc369d41e92f678b17ec27378fa5bf0576aeef7ac15460c13629ab5a}
+    command: [server, /data, --console-address, ':9001']
+    environment:
+      MINIO_ROOT_USER: admin
+      MINIO_ROOT_PASSWORD: password
+    volumes:
+      - minio-data:/data
+    ports:
+      - '127.0.0.1:${MINIO_CONSOLE_PORT:-9001}:9001'
+
+  # One-shot job: wait for MinIO, then create the warehouse bucket if absent.
+  minio-init:
+    image: 
${MINIO_CLIENT_IMAGE:-cgr.dev/chainguard/minio-client:latest-dev@sha256:fe649f71872c299489deb3cb9e057e5d2f76a45c85b7599f7c21616ac80b5e63}
+    depends_on:
+      minio:
+        condition: service_started
+    entrypoint: [/bin/sh, -ec]
+    command:
+      - |
+        for attempt in $$(seq 1 60); do
+          if mc alias set local http://minio:9000 admin password; then
+            mc mb --ignore-existing local/warehouse
+            exit 0
+          fi
+          sleep 2
+        done
+        echo 'MinIO did not become ready' >&2
+        exit 1
+
+  # Stores database/table schemas, locations and partition metadata in 
persistent
+  # Derby storage. Serves the internal Hive Metastore Thrift endpoint on port 
9083.
+  hive-metastore:
+    image: apache/hive:3.1.3
+    # The existing local-dev HMS image is amd64-only (emulated on ARM).
+    platform: linux/amd64
+    depends_on:
+      minio-init:
+        condition: service_completed_successfully
+    user: '0:0'
+    entrypoint: [/bin/bash, /lakehouse/start-metastore.sh]
+    environment:
+      SERVICE_NAME: metastore
+    volumes:
+      - metastore-data:/opt/hive/data
+      # Shared schema/server settings, including HMS's own S3A access to MinIO.
+      - ./conf/hive-site.xml:/opt/hive/conf/hive-site.xml:ro
+      - ./scripts/start-metastore.sh:/lakehouse/start-metastore.sh:ro
+    healthcheck:
+      test: [CMD, bash, -c, 'exec 3<>/dev/tcp/127.0.0.1/9083']
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 30s
+
+  # Executes SQL with an HMS-backed HoodieCatalog and reads/writes Hudi data in
+  # MinIO. Runs local[2]; external Connect clients use localhost:15002 by 
default.
+  spark-connect:
+    image: ${SPARK_CONNECT_IMAGE:-hudi-lakehouse-spark-connect:4.1.3-hudi1.2.0}
+    depends_on:
+      hive-metastore:
+        condition: service_healthy
+    environment:
+      AWS_ACCESS_KEY_ID: admin
+      AWS_SECRET_ACCESS_KEY: password
+      AWS_REGION: us-east-1
+    volumes:
+      - ./conf/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
+      - ./scripts/check-connect.py:/lakehouse/check-connect.py:ro
+    ports:
+      - '127.0.0.1:${SPARK_CONNECT_PORT:-15002}:15002'
+    healthcheck:
+      test: [CMD, python3, /lakehouse/check-connect.py]
+      interval: 30s
+      timeout: 30s
+      retries: 20
+      start_period: 30s
+    stop_grace_period: 30s
+
+  # One-shot Python Connect client: create/populate trips and drivers, then 
verify
+  # the sample data. Reuses the Spark image without starting another Spark 
server.
+  seed:
+    image: ${SPARK_CONNECT_IMAGE:-hudi-lakehouse-spark-connect:4.1.3-hudi1.2.0}
+    depends_on:
+      spark-connect:
+        condition: service_healthy
+    entrypoint: [python3, /lakehouse/seed.py]

Review Comment:
   `seed.py` runs as PID 1 here, where the kernel ignores a default-action 
SIGALRM, so `signal.alarm(600)` never fires and a stalled seed is never 
stopped. Adding `init: true` to the `seed` service lets the alarm terminate it 
(exit 142).



##########
hudi-lakehouse/docker/lakehouse.sh:
##########
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ACTION=
+BUILD=0
+LOCAL_LLM=0
+CHAT_CONFIG_ERROR=
+usage() {
+  cat <<'HELP'
+Usage: lakehouse.sh <up|down|status|logs|test> [--build] [--local-llm]
+  up             Start services, seed demo data, verify MCP and report chat 
readiness.
+  --build        Build Spark Connect and Gateway images in Docker from this 
checkout.
+  --local-llm    Start containerized Ollama and pull GATEWAY_LLM_MODEL 
(default qwen3:8b).
+  test           Verify sample data and gateway MCP without reseeding.
+  down           Remove containers/network; preserve all named data volumes.
+  status|logs    Inspect this Compose installation.
+  -h, --help     Show this help (also shown when no arguments are supplied).
+Set COMPOSE_PROJECT_NAME to isolate an installation; configure ports/provider 
in .env.
+An unavailable chat LLM does not fail startup; MCP clients use their own model.
+HELP
+}
+if [[ $# -eq 0 ]]; then
+  usage
+  exit 0
+fi
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    up|down|status|logs|test)
+      if [[ -n "$ACTION" ]]; then
+        echo 'Specify exactly one command.' >&2; usage >&2; exit 2
+      fi
+      ACTION="$1" ;;
+    --build) BUILD=1 ;;
+    --local-llm) LOCAL_LLM=1 ;;
+    -h|--help) usage; exit 0 ;;
+    *) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;;
+  esac
+  shift
+done
+if [[ -z "$ACTION" ]]; then
+  echo 'Specify a command, for example: lakehouse.sh up' >&2
+  usage >&2
+  exit 2
+fi
+command -v docker >/dev/null || { echo 'Install Docker with Compose 2.27.2 or 
newer first.' >&2; exit 1; }
+COMPOSE_VERSION="$(docker compose version --short)" || {
+  echo 'Docker Compose 2.27.2 or newer is required.' >&2; exit 1;
+}
+# config --environment was introduced in 2.27.2. Allow Docker Desktop suffixes.
+if [[ ! "$COMPOSE_VERSION" =~ ^v?([0-9]+)\.([0-9]+)\.([0-9]+)([-+].*)?$ ]] ||
+    (( 10#${BASH_REMATCH[1]} < 2 ||
+       (10#${BASH_REMATCH[1]} == 2 && (10#${BASH_REMATCH[2]} < 27 ||
+        (10#${BASH_REMATCH[2]} == 27 && 10#${BASH_REMATCH[3]} < 2))) )); then
+  echo "Docker Compose 2.27.2 or newer is required; found '$COMPOSE_VERSION'. 
Upgrade Docker Compose and retry." >&2
+  exit 1
+fi
+MEMORY="$(docker info --format '{{.MemTotal}}')" || {
+  echo 'Docker is not running or is inaccessible. Start Docker and retry.' 
>&2; exit 1;
+}
+COMPOSE=(docker compose --project-directory "$HERE" -f "$HERE/compose.yaml")
+if [[ -f "$HERE/.env" ]]; then
+  COMPOSE+=(--env-file "$HERE/.env")
+fi
+if [[ "$BUILD" == 1 ]]; then
+  COMPOSE+=(-f "$HERE/compose.build.yaml")
+fi
+if [[ "$LOCAL_LLM" == 1 ]]; then
+  export GATEWAY_LLM_PROVIDER=ollama 
GATEWAY_OLLAMA_BASE_URL=http://ollama:11434
+  COMPOSE+=(--profile local-llm)
+fi
+case "$ACTION" in
+  down) "${COMPOSE[@]}" --profile local-llm down; exit ;;
+  status) "${COMPOSE[@]}" --profile local-llm ps -a; exit ;;
+  logs) "${COMPOSE[@]}" --profile local-llm logs --tail=100; exit ;;
+esac
+if [[ "$ACTION" == up ]]; then
+  if [[ ! "$MEMORY" =~ ^[0-9]+$ ]] || (( MEMORY < 7500000000 )); then
+    echo 'Allocate at least 8 GB of memory to Docker (more for containerized 
Ollama).' >&2
+    exit 1
+  fi
+  # Auto-select API credentials from the environment only when the user has not
+  # supplied a .env configuration or an explicit provider. Never persist API 
keys.
+  if [[ ! -f "$HERE/.env" && -z "${GATEWAY_LLM_PROVIDER:-}" ]]; then

Review Comment:
   With `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` exported for other tools and no 
`.env`, `up` silently moves chat off the documented host-Ollama default, and 
once `GATEWAY_LLM_MODEL` names an Ollama model nothing flags it while every 
chat request fails. Is the key-based auto-selection needed, or could the 
installer at least print the provider and model it resolved and say it was 
auto-selected?



##########
hudi-lakehouse/docker/scripts/seed.py:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Seed the demo tables through Spark Connect; retries never replace existing 
rows."""
+import argparse
+import signal
+
+from pyspark.sql import SparkSession
+
+
+def seed(spark):
+    spark.sql("""CREATE TABLE IF NOT EXISTS default.drivers (
+        driver_id STRING, driver_name STRING, home_city STRING,
+        vehicle_type STRING, onboarded_date DATE
+    ) USING hudi TBLPROPERTIES (type = 'cow', primaryKey = 
'driver_id')""").collect()
+    spark.sql("""CREATE TABLE IF NOT EXISTS default.trips (
+        trip_id STRING, rider_id STRING, driver_id STRING, city STRING,
+        trip_status STRING, fare_amount DECIMAL(10,2), tip_amount 
DECIMAL(10,2),
+        distance_km DOUBLE, pickup_ts TIMESTAMP, dropoff_ts TIMESTAMP, 
trip_date DATE
+    ) USING hudi PARTITIONED BY (trip_date)
+    TBLPROPERTIES (type = 'cow', primaryKey = 'trip_id', preCombineField = 
'pickup_ts')""").collect()
+    spark.sql("""INSERT INTO default.drivers
+        SELECT source.* FROM (
+        SELECT * FROM VALUES
+          ('driver_1', 'Ananya', 'chennai', 'sedan', DATE '2024-01-01'),
+          ('driver_2', 'Alex', 'san_francisco', 'sedan', DATE '2024-01-01'),
+          ('driver_3', 'Ana', 'sao_paulo', 'sedan', DATE '2024-01-01')
+        AS drivers(driver_id, driver_name, home_city, vehicle_type, 
onboarded_date)
+    ) source LEFT ANTI JOIN default.drivers target
+        ON target.driver_id = source.driver_id""").collect()
+    # Stable keys make reruns safe, including a retry after only one table 
committed.
+    # First-run dates are yesterday; subsequent installs preserve the original 
data.
+    spark.sql("""INSERT INTO default.trips
+        SELECT source.* FROM (
+        SELECT concat('trip_', CAST(id AS STRING)) AS trip_id,
+               concat('rider_', CAST(id AS STRING)) AS rider_id,
+               concat('driver_', CAST(CAST(id / 3 AS INT) + 1 AS STRING)) AS 
driver_id,
+               element_at(array('chennai', 'san_francisco', 'sao_paulo'),
+                          CAST(id / 3 AS INT) + 1) AS city,
+               CASE WHEN id % 3 = 2 THEN 'cancelled' ELSE 'completed' END AS 
trip_status,
+               CAST(10 * (CAST(id / 3 AS INT) + 1) + (id % 3) * 10 AS 
DECIMAL(10,2)) AS fare_amount,
+               CAST(CAST(id / 3 AS INT) + 1 + id % 3 AS DECIMAL(10,2)) AS 
tip_amount,
+               CAST(5 + id AS DOUBLE) AS distance_km,
+               CAST(date_sub(current_date(), 1) AS TIMESTAMP) AS pickup_ts,
+               CAST(date_sub(current_date(), 1) AS TIMESTAMP) + INTERVAL 15 
MINUTES AS dropoff_ts,
+               date_sub(current_date(), 1) AS trip_date
+        FROM range(9)
+    ) source LEFT ANTI JOIN default.trips target
+        ON target.trip_id = source.trip_id""").collect()
+
+
+def verify(spark):
+    assert spark.conf.get("spark.sql.catalog.spark_catalog") == (
+        "org.apache.spark.sql.hudi.catalog.HoodieCatalog"
+    )
+    tables = {row.tableName for row in spark.sql("SHOW TABLES IN 
spark_catalog.default").collect()}
+    assert {"trips", "drivers"} <= tables, tables
+    assert spark.sql("SELECT count(*) FROM default.trips").collect()[0][0] == 9

Review Comment:
   Every `up` reruns the seed and the Gateway waits for it, so editing a demo 
row through the advertised Spark Connect endpoint (an `UPDATE` on `trips`, say) 
makes every later `up` fail here, and after a `down` the Gateway and MCP no 
longer start. Is that intended, or should a mismatch on tables that already 
existed only warn and leave the strict check to `lakehouse.sh test`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to