This is an automated email from the ASF dual-hosted git repository.
vinishjail97 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
The following commit(s) were added to refs/heads/main by this push:
new 2ba6faa4 [727] Fix the demo so it runs against current main, and
harden the image (#850)
2ba6faa4 is described below
commit 2ba6faa4f0bf799182151821e2bfbef72e8bd1ed
Author: Ranga Reddy <[email protected]>
AuthorDate: Fri Aug 7 04:02:01 2026 +0530
[727] Fix the demo so it runs against current main, and harden the image
(#850)
The demo had stopped working: the notebooks pinned dependency versions that
no
longer matched what xtable is built against, and the first cell provisioned
them
over the network on every kernel start. This makes the demo track the root
pom,
bakes the dependencies into the image, and removes the foot-guns that let
it rot
unnoticed in the first place.
Demo runtime:
- Derive every dependency version from the root pom. build_demo.sh writes
demo/jars/versions.properties, and both the image's pre-fetch and the
kernel
predef read it, so there is one place a version is stated and the demo
cannot
silently drift from what xtable is tested against.
- Bake JDK 11 and all notebook dependencies into the Jupyter image. Hudi 1.x
requires Java 11+ while the almond base image only ships Java 8, and the
coursier cache is warmed at build time so the first cell does not spend
minutes resolving.
- Move dependency loading out of the notebooks into a kernel predef, so the
notebooks contain no dependency management.
- Gate the Jupyter startup on the dependencies being resolvable, and
re-fetch
anything missing if versions changed since the image was built.
- Run the demo in the background, split the jar build out into
build_demo.sh,
and add stop_demo.sh with --reset-data to restore the seed datasets.
- Avoid an in-place Hudi table upgrade on the seed table, and make the Delta
join cell re-runnable so the notebook can be executed more than once.
- Create the demo namespace in the metastore automatically, qualifying the
array
construction as scala.Array: the notebook imports java.util._, so a bare
Array(namespace) resolves against java.lang.reflect.Array in Ammonite.
- Silence the harmless Presto RMI accept-loop warning via a log.properties.
Hardening, from review on the PR:
- Publish the notebook server on 127.0.0.1 only. It runs with authentication
disabled, and 8888 was bound to every interface, so anyone able to reach
the
host had an unauthenticated Scala kernel: arbitrary code execution with
read/write access to the mounted demo/data. Loopback keeps the printed
http://localhost:8888/lab URL working. The other three services are left
alone; they are pre-existing and not unauthenticated code execution.
- Pin the base image to almondsh/almond:0.14.1 by tag and digest. The
kernel-patching step depends on the base image's kernel path, on argv[0]
being
"java" and on -Xmx4g being safe to insert at argv[1], so a moving :latest
could break the demo at build time - the same class of failure this change
exists to fix. 0.14.1 is what :latest resolved to, same manifest digest,
so
behaviour is unchanged.
- Assert in the kernel-patching step instead of silently doing nothing.
With no
kernel matched the glob returned an empty list, the comprehension iterated
over nothing, and the build succeeded with an unpatched image that fails
only
when a notebook is run. Both the kernel path and the argv shape are
checked.
- Read the log4j, trino-jdbc and presto-jdbc versions from
versions.properties
like every other coordinate. These four were hardcoded in both
prefetch_dependencies.sh and predef.sc, and a divergence between the two
lists
silently sends every kernel start to the network instead of the pre-warmed
cache. log4j moves from 2.17.2 to the root pom's 2.22.0; 2.17.2 was
carried
over from the original demo (5d40c6f), not required by almond or Spark.
- Fail in prefetch_dependencies.sh when a property is missing, rather than
building a coordinate like "org.apache.spark:spark-sql_:" and surfacing
it as
a coursier resolution error. Values are read with cut -f2- so a value
containing "=" is not truncated.
- Remove stale xtable jars from demo/jars before copying new ones. predef.sc
puts every jar in that directory on the kernel classpath, so after a
version
bump two copies of the same classes were loaded.
Verified: image builds with the pinned base and patches all three Scala
kernels;
the kernel starts and reports log4j 2.22.0, Java 11 and Spark 3.4.2 with the
xtable classes loaded; the notebook port is reachable on 127.0.0.1 and
refused
on the host's LAN address; both new asserts and the missing-property check
fail
loudly when their assumption is broken.
Closes #727
---
.gitignore | 3 +
.../catalog/iceberg.properties => .dockerignore} | 13 +-
demo/README.md | 7 +-
demo/build_demo.sh | 68 +++++++
demo/docker-compose.yaml | 13 +-
demo/jupyter/Dockerfile | 64 ++++++
demo/jupyter/predef.sc | 69 +++++++
demo/jupyter/prefetch_dependencies.sh | 70 +++++++
demo/jupyter/verify_dependencies.sh | 35 ++++
demo/notebook/demo.ipynb | 222 +++++++++++++++------
.../delta.properties => presto/log.properties} | 12 +-
demo/start_demo.sh | 49 ++++-
demo/{start_demo.sh => stop_demo.sh} | 20 +-
demo/trino/catalog/delta.properties | 3 +-
demo/trino/catalog/iceberg.properties | 3 +-
website/docs/demo/docker.md | 12 ++
16 files changed, 564 insertions(+), 99 deletions(-)
diff --git a/.gitignore b/.gitignore
index d62030a3..4273535c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -50,3 +50,6 @@ spec/generated
# SDKMan
.sdkmanrc
+demo/notebook/spark-warehouse/
+demo/notebook/metastore_db/
+demo/notebook/derby.log
diff --git a/demo/trino/catalog/iceberg.properties b/demo/.dockerignore
similarity index 80%
copy from demo/trino/catalog/iceberg.properties
copy to demo/.dockerignore
index d59b6861..b1e99cdf 100644
--- a/demo/trino/catalog/iceberg.properties
+++ b/demo/.dockerignore
@@ -14,9 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-connector.name=iceberg
-hive.metastore.uri=thrift://hive-metastore:9083
-hive.metastore-cache-ttl=0s
-hive.metastore-refresh-interval=5s
-hive.metastore-timeout=10s
-iceberg.catalog.type=hive_metastore
\ No newline at end of file
+## Only jars/versions.properties and jupyter/ are needed in the image build
+## context; everything else is mounted at runtime.
+data
+notebook
+presto
+trino
+jars/*.jar
diff --git a/demo/README.md b/demo/README.md
index eaada899..777aa381 100644
--- a/demo/README.md
+++ b/demo/README.md
@@ -19,7 +19,12 @@
# Running a Local Demo
This demo was created for the 2023 Open Source Data Summit. It shows how
XTable can be used with two existing datasets.
-Use `./start_demo.sh` to spin up a local notebook with a scala interpreter,
Hive Metastore, Presto and Trino in docker containers. The script will first
build the XTable jars required for the demo and then start the containers.
+Use `./start_demo.sh` to spin up a local notebook with a scala interpreter,
Hive Metastore, Presto and Trino in docker containers (in the background). The
script builds the XTable jars required for the demo first (if they are not
already present) and then starts the containers and prints the notebook URL.
The jars can also be (re)built explicitly with `./build_demo.sh`, for example
after making code changes.
+
+When you are done, use `./stop_demo.sh` to stop and remove the containers.
Pass `--reset-data` to also restore the seed datasets under `demo/data` to
their original state so the demo can be re-run from scratch.
+
+## Notebooks
+- `work/demo.ipynb` — the main demo: two existing datasets (Hudi and Delta)
are synced across Hudi, Delta and Iceberg, updated from Trino, joined, and
validated from Trino.
## Accessing Services
### Jupyter Notebook
diff --git a/demo/build_demo.sh b/demo/build_demo.sh
new file mode 100755
index 00000000..5e878377
--- /dev/null
+++ b/demo/build_demo.sh
@@ -0,0 +1,68 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+## Build the XTable jars required for the demo and generate the dependency
+## version properties consumed by the demo notebooks.
+
+set -e
+
+CURRENT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
+XTABLE_HOME="$( cd "$(dirname "$CURRENT_DIR")" ; pwd -P )"
+cd "$XTABLE_HOME"
+
+## Read a property from the root pom so the demo always matches the versions
+## xtable is built against.
+get_property() {
+ ./mvnw -q help:evaluate -Dexpression="$1" -DforceStdout
+}
+
+PROJECT_VERSION=$(get_property project.version)
+SCALA_BINARY_VERSION=$(get_property scala.binary.version)
+
+## JDBC drivers for the query engines the demo talks to. These have no root-pom
+## property because xtable does not depend on them, so they are declared here
to
+## keep versions.properties the single source for every coordinate the
notebooks
+## resolve. Keep them in step with the trino/presto image tags in
+## demo/docker-compose.yaml.
+TRINO_JDBC_VERSION=431
+PRESTO_JDBC_VERSION=0.283
+
+./mvnw install -am -pl xtable-core -DskipTests -T 2
+
+mkdir -p demo/jars
+## Drop xtable jars from a previous build first. predef.sc puts every jar in
+## demo/jars on the kernel classpath, so leaving an older version behind after
a
+## version bump loads two copies of the same classes.
+rm -f demo/jars/xtable-*.jar
+cp
"xtable-hudi-support/xtable-hudi-support-utils/target/xtable-hudi-support-utils-${PROJECT_VERSION}.jar"
demo/jars
+cp "xtable-api/target/xtable-api-${PROJECT_VERSION}.jar" demo/jars
+cp
"xtable-core/target/xtable-core_${SCALA_BINARY_VERSION}-${PROJECT_VERSION}.jar"
demo/jars
+
+## Export the dependency versions used by the notebooks. The notebooks read
+## this file from /home/jars/versions.properties instead of hardcoding
versions.
+{
+ cat style/text-license-header
+ echo "scala.binary.version=${SCALA_BINARY_VERSION}"
+ for prop in spark.version spark.version.prefix hudi.version delta.version
delta.kernel.version iceberg.hive.runtime.version log4j.version; do
+ echo "${prop}=$(get_property "${prop}")"
+ done
+ echo "trino.jdbc.version=${TRINO_JDBC_VERSION}"
+ echo "presto.jdbc.version=${PRESTO_JDBC_VERSION}"
+} > demo/jars/versions.properties
+
+echo "XTable demo jars and versions.properties generated under demo/jars:"
+ls demo/jars
diff --git a/demo/docker-compose.yaml b/demo/docker-compose.yaml
index beeabaa9..c32a325e 100644
--- a/demo/docker-compose.yaml
+++ b/demo/docker-compose.yaml
@@ -37,6 +37,7 @@ services:
- ./presto/config.properties:/opt/presto-server/etc/config.properties
- ./presto/jvm.config:/opt/presto-server/etc/jvm.config
- ./presto/node.properties:/opt/presto-server/etc/node.properties
+ - ./presto/log.properties:/opt/presto-server/etc/log.properties
- ./data:/home/data
hive-metastore:
@@ -54,9 +55,17 @@ services:
jupyter:
container_name: jupyter
hostname: jupyter
- image: 'almondsh/almond:latest'
+ build:
+ context: .
+ dockerfile: jupyter/Dockerfile
+ image: 'xtable-demo-jupyter'
+ ## Local demo only: the notebook server runs without authentication, so the
+ ## port is published to the loopback interface only. A Scala kernel is
+ ## arbitrary code execution with read/write access to the mounted ./data,
so
+ ## do not widen this to 0.0.0.0 without restoring the token.
+ command: ["start-notebook.sh", "--IdentityProvider.token=",
"--ServerApp.password="]
ports:
- - '8888:8888'
+ - '127.0.0.1:8888:8888'
volumes:
- ./notebook:/home/jovyan/work
- ./jars:/home/jars
diff --git a/demo/jupyter/Dockerfile b/demo/jupyter/Dockerfile
new file mode 100644
index 00000000..4ae229b7
--- /dev/null
+++ b/demo/jupyter/Dockerfile
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Jupyter image for the XTable demo, based on the almond Scala kernel image.
+# Everything the notebooks need at runtime is baked in at build time so the
+# first cell does not spend minutes provisioning:
+# - OpenJDK 11 for the Scala kernels (Hudi 1.x jars require Java 11+)
+# - all notebook dependencies (Spark, Hudi, Delta, Iceberg, JDBC drivers)
+# pre-fetched into the coursier cache
+# - the Scala kernels re-pointed at that JDK, with a bounded heap and a
+# predef that puts every dependency on the classpath at kernel startup
+# Pinned by tag and digest. The kernel-patching step at the bottom of this file
+# depends on the base image's layout - the kernel path, argv[0] being "java"
and
+# -Xmx4g being safe to insert at argv[1] - so a moving :latest could break the
+# demo silently at build time. 0.14.1 is the image :latest resolved to when
this
+# was pinned, so behaviour is unchanged; bump both tag and digest together.
+FROM
almondsh/almond:0.14.1@sha256:0407359be58529eaa8e35bcf4586c3a890123be64005600b381de88fdc3c5de7
+
+# Install a JDK 11 for the Scala kernels; the base image only ships Java 8.
+RUN mamba create -y -q -n jdk11 -c conda-forge openjdk=11 && mamba clean -afy
+
+# Pre-fetch the notebook dependencies into the coursier cache. The versions
+# come from demo/jars/versions.properties, generated from the root pom by
+# build_demo.sh, so the image always matches what xtable is built against.
+COPY jars/versions.properties /usr/local/share/xtable/versions.properties
+COPY jupyter/prefetch_dependencies.sh
/usr/local/share/xtable/prefetch_dependencies.sh
+RUN bash /usr/local/share/xtable/prefetch_dependencies.sh
/usr/local/share/xtable/versions.properties
+
+# Startup gate: re-verify (and complete, if versions changed since the image
+# was built) the dependency cache before the Jupyter server is started.
+COPY jupyter/verify_dependencies.sh
/usr/local/bin/before-notebook.d/10-xtable-verify-dependencies.sh
+
+# The kernel predef loads all notebook dependencies at kernel startup, so the
+# notebooks contain no dependency management code.
+COPY jupyter/predef.sc /usr/local/share/xtable/predef.sc
+
+# Point all Scala kernels at the JDK 11 with a bounded heap (local Spark for
+# the demo fits comfortably in 4g and this keeps the kernel JVM from starving
+# the other demo containers) and wire in the predef.
+#
+# The assertions matter: without them a base image whose kernel path or argv
+# shape changed would make the glob match nothing, the comprehension iterate
+# over an empty list, and the build succeed with an unpatched image that fails
+# only when a notebook is run.
+RUN python3 -c "import glob, json; \
+paths =
sorted(glob.glob('/home/jovyan/.local/share/jupyter/kernels/scala*/kernel.json'));
\
+assert paths, 'no scala kernels found under
/home/jovyan/.local/share/jupyter/kernels'; \
+kernels = [(p, json.load(open(p))) for p in paths]; \
+assert all(k['argv'][0] == 'java' for _, k in kernels), 'kernel argv[0] is not
java: %r' % [k['argv'][0] for _, k in kernels]; \
+[(k['argv'].__setitem__(0, '/opt/conda/envs/jdk11/bin/java'),
k['argv'].insert(1, '-Xmx4g'), k['argv'].extend(['--predef',
'/usr/local/share/xtable/predef.sc']), k.setdefault('env',
{}).__setitem__('JAVA_HOME', '/opt/conda/envs/jdk11'), json.dump(k, open(p,
'w'), indent=2)) for p, k in kernels]; \
+print('patched kernels:', [p for p, _ in kernels])"
diff --git a/demo/jupyter/predef.sc b/demo/jupyter/predef.sc
new file mode 100644
index 00000000..de9be73b
--- /dev/null
+++ b/demo/jupyter/predef.sc
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Kernel predef for the XTable demo notebooks: loads every dependency the
+// notebooks need when the kernel starts, so the notebooks themselves contain
+// no dependency management. The artifacts are already present in the image's
+// coursier cache (pre-fetched at image build time); versions come from
+// /home/jars/versions.properties, generated from the root pom by
+// build_demo.sh.
+
+val xtableVersions = {
+ val props = new java.util.Properties()
+ val in = new java.io.FileInputStream("/home/jars/versions.properties")
+ try props.load(in)
+ finally in.close()
+ props
+}
+
+def xtableVersion(key: String): String =
+
Option(xtableVersions.getProperty(key)).map(_.trim).filter(_.nonEmpty).getOrElse
{
+ sys.error(s"$key missing from /home/jars/versions.properties, re-run
build_demo.sh")
+ }
+
+{
+ val scalaBinaryVersion = xtableVersion("scala.binary.version")
+ val sparkVersionPrefix = xtableVersion("spark.version.prefix")
+
+ // Keep this list resolving to the same artifacts as the coursier fetch in
+ // prefetch_dependencies.sh; a divergence means a network resolve on every
+ // kernel start instead of a cache hit.
+ Seq(
+ ("org.apache.logging.log4j", "log4j-api", xtableVersion("log4j.version")),
+ ("org.apache.logging.log4j", "log4j-core", xtableVersion("log4j.version")),
+ ("org.apache.spark", s"spark-sql_$scalaBinaryVersion",
xtableVersion("spark.version")),
+ ("org.apache.spark", s"spark-hive_$scalaBinaryVersion",
xtableVersion("spark.version")),
+ ("org.apache.hudi",
s"hudi-spark$sparkVersionPrefix-bundle_$scalaBinaryVersion",
xtableVersion("hudi.version")),
+ ("org.apache.hudi", "hudi-java-client", xtableVersion("hudi.version")),
+ ("io.delta", s"delta-core_$scalaBinaryVersion",
xtableVersion("delta.version")),
+ ("io.delta", "delta-kernel-api", xtableVersion("delta.kernel.version")),
+ ("io.delta", "delta-kernel-defaults",
xtableVersion("delta.kernel.version")),
+ ("org.apache.iceberg", "iceberg-hive-runtime",
xtableVersion("iceberg.hive.runtime.version")),
+ ("io.trino", "trino-jdbc", xtableVersion("trino.jdbc.version")),
+ ("com.facebook.presto", "presto-jdbc",
xtableVersion("presto.jdbc.version"))
+ ).foreach { case (org, name, version) =>
+ interp.load.ivy(coursierapi.Dependency.of(org, name, version))
+ }
+
+ // Load the xtable jars built by build_demo.sh (version-agnostic).
+ new java.io.File("/home/jars").listFiles
+ .filter(_.getName.endsWith(".jar"))
+ .foreach(jar => interp.load.cp(os.Path(jar.getAbsolutePath)))
+}
+
+println("XTable demo kernel ready: Spark, Hudi, Delta, Iceberg and the XTable
jars are on the classpath.")
diff --git a/demo/jupyter/prefetch_dependencies.sh
b/demo/jupyter/prefetch_dependencies.sh
new file mode 100644
index 00000000..f9160b98
--- /dev/null
+++ b/demo/jupyter/prefetch_dependencies.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+## Pre-fetch the demo notebook dependencies into the coursier cache at image
+## build time so the notebooks' first cell resolves instantly. The dependency
+## versions are read from the versions.properties file passed as the first
+## argument (generated from the root pom by build_demo.sh).
+set -e
+
+PROPS_FILE="$1"
+
+## Fail on a missing or renamed key rather than substituting an empty string:
+## otherwise a typo, or build_demo.sh dropping a property, produces a
coordinate
+## like "org.apache.spark:spark-sql_:" and a coursier resolution error that
says
+## nothing about the real cause. Mirrors what predef.sc does.
+prop() {
+ local value
+ value=$(grep "^$1=" "${PROPS_FILE}" | cut -d= -f2-)
+ if [ -z "${value}" ]; then
+ echo "ERROR: $1 missing from ${PROPS_FILE}; re-run build_demo.sh" >&2
+ exit 1
+ fi
+ echo "${value}"
+}
+
+SCALA_BINARY_VERSION=$(prop scala.binary.version)
+SPARK_VERSION=$(prop spark.version)
+SPARK_VERSION_PREFIX=$(prop spark.version.prefix)
+HUDI_VERSION=$(prop hudi.version)
+DELTA_VERSION=$(prop delta.version)
+DELTA_KERNEL_VERSION=$(prop delta.kernel.version)
+ICEBERG_HIVE_RUNTIME_VERSION=$(prop iceberg.hive.runtime.version)
+LOG4J_VERSION=$(prop log4j.version)
+TRINO_JDBC_VERSION=$(prop trino.jdbc.version)
+PRESTO_JDBC_VERSION=$(prop presto.jdbc.version)
+
+## Every coordinate's version comes from versions.properties. This list and the
+## one in predef.sc must resolve to the same artifacts: this one warms the
+## coursier cache at image build time, that one resolves at kernel startup, so
a
+## divergence of one version sends every kernel start back to the network -
+## exactly the slow path this image exists to avoid.
+coursier fetch --quiet \
+ "org.apache.logging.log4j:log4j-api:${LOG4J_VERSION}" \
+ "org.apache.logging.log4j:log4j-core:${LOG4J_VERSION}" \
+ "org.apache.spark:spark-sql_${SCALA_BINARY_VERSION}:${SPARK_VERSION}" \
+ "org.apache.spark:spark-hive_${SCALA_BINARY_VERSION}:${SPARK_VERSION}" \
+
"org.apache.hudi:hudi-spark${SPARK_VERSION_PREFIX}-bundle_${SCALA_BINARY_VERSION}:${HUDI_VERSION}"
\
+ "org.apache.hudi:hudi-java-client:${HUDI_VERSION}" \
+ "io.delta:delta-core_${SCALA_BINARY_VERSION}:${DELTA_VERSION}" \
+ "io.delta:delta-kernel-api:${DELTA_KERNEL_VERSION}" \
+ "io.delta:delta-kernel-defaults:${DELTA_KERNEL_VERSION}" \
+ "org.apache.iceberg:iceberg-hive-runtime:${ICEBERG_HIVE_RUNTIME_VERSION}" \
+ "io.trino:trino-jdbc:${TRINO_JDBC_VERSION}" \
+ "com.facebook.presto:presto-jdbc:${PRESTO_JDBC_VERSION}"
+
+echo "Notebook dependencies pre-fetched into the coursier cache."
diff --git a/demo/jupyter/verify_dependencies.sh
b/demo/jupyter/verify_dependencies.sh
new file mode 100644
index 00000000..6e942e0a
--- /dev/null
+++ b/demo/jupyter/verify_dependencies.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+## Startup gate for the Jupyter container: verifies that every notebook
+## dependency is present in the local coursier cache (downloading anything
+## missing, e.g. after a version bump without an image rebuild) BEFORE the
+## Jupyter server starts. The image build pre-fetches everything, so when the
+## versions are unchanged this completes in a few seconds without any network
+## access.
+
+PROPS_FILE=/home/jars/versions.properties
+if [ ! -f "${PROPS_FILE}" ]; then
+ PROPS_FILE=/usr/local/share/xtable/versions.properties
+fi
+
+echo "xtable-demo: verifying notebook dependencies before starting Jupyter
(using ${PROPS_FILE})..."
+if ! bash /usr/local/share/xtable/prefetch_dependencies.sh "${PROPS_FILE}";
then
+ echo "xtable-demo: ERROR - failed to resolve the notebook dependencies; not
starting Jupyter." >&2
+ exit 1
+fi
+echo "xtable-demo: all notebook dependencies are available, starting Jupyter."
diff --git a/demo/notebook/demo.ipynb b/demo/notebook/demo.ipynb
index 3538ed3e..7ce957cb 100644
--- a/demo/notebook/demo.ipynb
+++ b/demo/notebook/demo.ipynb
@@ -5,33 +5,21 @@
"id": "29a0a252-5a8a-4986-af24-1e2af6dfa081",
"metadata": {},
"source": [
- "# Pre-Demo Setup"
+ "# Pre-Demo Setup\n",
+ "All required dependencies (Spark, Hudi, Delta Lake, Iceberg and the
XTable jars) are loaded automatically\n",
+ "when the kernel starts, so we begin directly with the imports."
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "5f02fa4f-ca2b-49da-b2e1-9361bf001633",
- "metadata": {
- "scrolled": true
- },
+ "id": "afcc578a",
+ "metadata": {},
"outputs": [],
"source": [
- "import $ivy.`org.apache.logging.log4j:log4j-api:2.17.2`\n",
- "import $ivy.`org.apache.logging.log4j:log4j-core:2.17.2`\n",
- "import $ivy.`org.apache.spark::spark-sql:3.4.2`\n",
- "import $ivy.`org.apache.spark:spark-hive_2.12:3.4.2`\n",
"import org.apache.spark.sql._\n",
"import org.apache.log4j.{Level, Logger}\n",
"Logger.getLogger(\"org\").setLevel(Level.ERROR)\n",
- "import $ivy.`org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0`\n",
- "import $ivy.`org.apache.hudi:hudi-java-client:0.14.0`\n",
- "import $ivy.`io.delta:delta-core_2.12:2.4.0`\n",
- "import $cp.`/home/jars/xtable-core-0.5.0-SNAPSHOT.jar`\n",
- "import $cp.`/home/jars/xtable-api-0.5.0-SNAPSHOT.jar`\n",
- "import $cp.`/home/jars/xtable-hudi-support-utils-0.5.0-SNAPSHOT.jar`\n",
- "import $ivy.`org.apache.iceberg:iceberg-hive-runtime:1.4.2`\n",
- "import $ivy.`io.trino:trino-jdbc:431`\n",
"import java.util._\n",
"import org.apache.xtable.conversion._\n",
"import org.apache.xtable.model.sync._\n",
@@ -49,28 +37,29 @@
"source": [
"val hudiTableName = \"hudi_dimCustomer\"\n",
"val hudiBasePath = \"file:/home/data/db/hudi_dimCustomer\"\n",
- "val writerSchema =
\"{\\\"type\\\":\\\"record\\\",\\\"name\\\":\\\"Sample\\\",\\\"fields\\\":[{\\\"name\\\":\\\"_c0\\\",\\\"type\\\":\\\"string\\\"},{\\\"name\\\":\\\"CustomerKey\\\",\\\"type\\\":\\\"string\\\"},{\\\"name\\\":\\\"GeographyKey\\\",\\\"type\\\":\\\"string\\\"},{\\\"name\\\":\\\"FirstName\\\",\\\"type\\\":\\\"string\\\"},{\\\"name\\\":\\\"LastName\\\",\\\"type\\\":\\\"string\\\"},{\\\"name\\\":\\\"BirthDate\\\",\\\"type\\\":\\\"string\\\"},{\\\"name\\\":\\\"MaritalStat
[...]
+ "\n",
"val hudiWriteOptions = new HashMap[String, String]()\n",
"hudiWriteOptions.put(\"hoodie.table.name\", hudiTableName)\n",
"hudiWriteOptions.put(\"hoodie.datasource.write.recordkey.field\",
\"CustomerKey\")\n",
"hudiWriteOptions.put(\"hoodie.datasource.write.partitionpath.field\",
\"\")\n",
"hudiWriteOptions.put(\"hoodie.datasource.write.precombine.field\",
\"_c0\")\n",
"hudiWriteOptions.put(\"hoodie.datasource.write.operation\",
\"insert\")\n",
- "hudiWriteOptions.put(\"hoodie.write.schema\", writerSchema)\n",
"hudiWriteOptions.put(\"hoodie.populate.meta.fields\", \"false\")\n",
"hudiWriteOptions.put(\"hoodie.parquet.small.file.limit\", \"1\")\n",
+ "hudiWriteOptions.put(\"hoodie.write.table.version\", \"6\")\n",
"\n",
"val deltaTableName = \"delta_dimGeography\"\n",
"val deltaBasePath = \"file:/home/data/db/delta_dimGeography\"\n",
"\n",
"val spark = org.apache.spark.sql.SparkSession.builder()\n",
- " .appName(\"demo\")\n",
+ " .appName(\"XTable Docker Demo\")\n",
" .config(\"spark.serializer\",
\"org.apache.spark.serializer.KryoSerializer\")\n",
" .config(\"spark.sql.catalog.spark_catalog\",
\"org.apache.spark.sql.hudi.catalog.HoodieCatalog\")\n",
" .config(\"spark.sql.extensions\",
\"org.apache.spark.sql.hudi.HoodieSparkSessionExtension\")\n",
" .config(\"spark.kryo.registrator\",
\"org.apache.spark.HoodieSparkKryoRegistrar\")\n",
" .master(\"local\")\n",
" .getOrCreate()\n",
+ "\n",
"spark.sparkContext.setLogLevel(\"ERROR\")"
]
},
@@ -92,7 +81,9 @@
"id": "dd09e5d0-f71f-4000-89fe-b62fda7236da",
"metadata": {},
"source": [
- ""
+ "<p align=\"center\">\n",
+ " <img src=\"attachment:2af13f55-97a1-415f-a354-bdf065dbf3af.png\"
alt=\"image.png\"/>\n",
+ "</p>"
]
},
{
@@ -111,27 +102,57 @@
"metadata": {},
"outputs": [],
"source": [
- "val namespace = \"demo\" // requires that this db is already created in
your HMS, in Trino you can run `create schema hive.demo`\n",
- "val namespaceArray = Array(namespace)\n",
- "val catalogOptions = new HashMap[String, String]();\n",
- "catalogOptions.put(\"uri\", \"thrift://hive-metastore:9083\");\n",
+ "val namespace = \"demo\" // created in the metastore below if it does not
exist yet\n",
+ "val namespaceArray = scala.Array(namespace)\n",
+ "\n",
+ "val catalogOptions = new HashMap[String, String]()\n",
+ "catalogOptions.put(\"uri\", \"thrift://hive-metastore:9083\")\n",
+ "catalogOptions.put(\"warehouse\", \"file:/home/data\")\n",
+ "\n",
"val icebergCatalogConfig = IcebergCatalogConfig.builder()\n",
" .catalogImpl(\"org.apache.iceberg.hive.HiveCatalog\")\n",
" .catalogName(\"iceberg\")\n",
" .catalogOptions(catalogOptions)\n",
- " .build();\n",
- "val perTableConfig = PerTableConfigImpl.builder()\n",
- " .tableName(hudiTableName)\n",
+ " .build()\n",
+ "\n",
+ "// Create the namespace in the metastore if it does not already exist, so
the\n",
+ "// demo works out of the box on a freshly started stack.\n",
+ "val hiveCatalog = new org.apache.iceberg.hive.HiveCatalog()\n",
+ "hiveCatalog.setConf(spark.sparkContext.hadoopConfiguration)\n",
+ "hiveCatalog.initialize(\"iceberg\", catalogOptions)\n",
+ "\n",
+ "val icebergNamespace =
org.apache.iceberg.catalog.Namespace.of(namespaceArray: _*)\n",
+ "if (!hiveCatalog.namespaceExists(icebergNamespace)) {\n",
+ " hiveCatalog.createNamespace(icebergNamespace)\n",
+ "}\n",
+ "\n",
+ "val sourceTable = SourceTable.builder()\n",
+ " .name(hudiTableName)\n",
+ " .formatName(TableFormat.HUDI)\n",
+ " .basePath(hudiBasePath)\n",
+ " .additionalProperties(new Properties())\n",
+ " .build()\n",
+ "\n",
+ "val icebergTargetTable = TargetTable.builder()\n",
+ " .name(hudiTableName)\n",
+ " .formatName(TableFormat.ICEBERG)\n",
+ " .basePath(hudiBasePath)\n",
" .namespace(namespaceArray)\n",
- " .targetTableFormats(Arrays.asList(TableFormat.ICEBERG))\n",
- " .tableBasePath(hudiBasePath)\n",
- " .icebergCatalogConfig(icebergCatalogConfig)\n",
+ " .catalogConfig(icebergCatalogConfig)\n",
+ " .additionalProperties(new Properties())\n",
+ " .build()\n",
+ "\n",
+ "val conversionConfig = ConversionConfig.builder()\n",
+ " .sourceTable(sourceTable)\n",
+ " .targetTables(Arrays.asList(icebergTargetTable))\n",
" .syncMode(SyncMode.INCREMENTAL)\n",
" .build()\n",
+ "\n",
"val hudiConversionSourceProvider = new HudiConversionSourceProvider()\n",
- "hudiConversionSourceProvider.init(spark.sparkContext.hadoopConfiguration,
Collections.emptyMap())\n",
+
"hudiConversionSourceProvider.init(spark.sparkContext.hadoopConfiguration)\n",
+ "\n",
"val conversionController = new
ConversionController(spark.sparkContext.hadoopConfiguration)\n",
- "conversionController.sync(perTableConfig, hudiConversionSourceProvider)"
+ "conversionController.sync(conversionConfig, hudiConversionSourceProvider)"
]
},
{
@@ -156,13 +177,15 @@
" .format(\"csv\")\n",
" .option(\"header\", \"true\")\n",
" .load(\"/home/data/DimCustomer_round2.csv\")\n",
+ "\n",
"hudiCsvSourcePart2\n",
" .write\n",
" .format(\"hudi\")\n",
" .options(hudiWriteOptions)\n",
" .mode(\"append\")\n",
" .save(hudiBasePath)\n",
- "conversionController.sync(perTableConfig, hudiConversionSourceProvider)"
+ "\n",
+ "conversionController.sync(conversionConfig, hudiConversionSourceProvider)"
]
},
{
@@ -182,11 +205,14 @@
"outputs": [],
"source": [
"import java.sql._\n",
+ "\n",
"val url = \"jdbc:trino://trino:8080/iceberg/\" + namespace +
\"?user=admin\"\n",
"val connection = DriverManager.getConnection(url)\n",
"val statement = connection.createStatement()\n",
+ "\n",
"statement.executeUpdate(\"INSERT INTO
hudi_dimCustomer(_c0,CustomerKey,GeographyKey,FirstName,LastName,BirthDate,MaritalStatus,Gender,YearlyIncome,TotalChildren,NumberChildrenAtHome,Education,Occupation,HouseOwnerFlag,NumberCarsOwned)
\" +\n",
" \"VALUES ('18869','19146','735','Reginald','Not
Provided','2020-07-04','S','M','1000000000.0','0.0','0.0','Puppy Obedience
School','Branch Manager','1.0','2.0')\")\n",
+ "\n",
"statement.close()\n",
"connection.close()"
]
@@ -208,16 +234,31 @@
"outputs": [],
"source": [
"val icebergConversionSourceProvider = new
IcebergConversionSourceProvider()\n",
-
"icebergConversionSourceProvider.init(spark.sparkContext.hadoopConfiguration,
Collections.emptyMap())\n",
- "val icebergSourcePerTableConfig = PerTableConfigImpl.builder()\n",
- " .tableName(hudiTableName)\n",
+
"icebergConversionSourceProvider.init(spark.sparkContext.hadoopConfiguration)\n",
+ "\n",
+ "val icebergSourceTable = SourceTable.builder()\n",
+ " .name(hudiTableName)\n",
+ " .formatName(TableFormat.ICEBERG)\n",
+ " .basePath(hudiBasePath)\n",
" .namespace(namespaceArray)\n",
- " .targetTableFormats(Arrays.asList(TableFormat.DELTA))\n",
- " .tableBasePath(hudiBasePath)\n",
- " .icebergCatalogConfig(icebergCatalogConfig)\n",
+ " .catalogConfig(icebergCatalogConfig)\n",
+ " .additionalProperties(new Properties())\n",
+ " .build()\n",
+ "\n",
+ "val deltaTargetTable = TargetTable.builder()\n",
+ " .name(hudiTableName)\n",
+ " .formatName(TableFormat.DELTA)\n",
+ " .basePath(hudiBasePath)\n",
+ " .additionalProperties(new Properties())\n",
+ " .build()\n",
+ "\n",
+ "val icebergSourceConversionConfig = ConversionConfig.builder()\n",
+ " .sourceTable(icebergSourceTable)\n",
+ " .targetTables(Arrays.asList(deltaTargetTable))\n",
" .syncMode(SyncMode.INCREMENTAL)\n",
" .build()\n",
- "conversionController.sync(icebergSourcePerTableConfig,
icebergConversionSourceProvider)"
+ "\n",
+ "conversionController.sync(icebergSourceConversionConfig,
icebergConversionSourceProvider)"
]
},
{
@@ -240,15 +281,31 @@
"source": [
"// Sync from a Delta Lake table to Hudi\n",
"import org.apache.xtable.delta._\n",
+ "\n",
"val deltaConversionSourceProvider = new
DeltaConversionSourceProvider()\n",
-
"deltaConversionSourceProvider.init(spark.sparkContext.hadoopConfiguration,
Collections.emptyMap())\n",
- "val deltaSourcePerTableConfig = PerTableConfigImpl.builder()\n",
- " .tableName(deltaTableName)\n",
- " .targetTableFormats(Arrays.asList(TableFormat.HUDI))\n",
- " .tableBasePath(deltaBasePath)\n",
+
"deltaConversionSourceProvider.init(spark.sparkContext.hadoopConfiguration)\n",
+ "\n",
+ "val deltaSourceTable = SourceTable.builder()\n",
+ " .name(deltaTableName)\n",
+ " .formatName(TableFormat.DELTA)\n",
+ " .basePath(deltaBasePath)\n",
+ " .additionalProperties(new Properties())\n",
+ " .build()\n",
+ "\n",
+ "val hudiTargetTable = TargetTable.builder()\n",
+ " .name(deltaTableName)\n",
+ " .formatName(TableFormat.HUDI)\n",
+ " .basePath(deltaBasePath)\n",
+ " .additionalProperties(new Properties())\n",
+ " .build()\n",
+ "\n",
+ "val deltaSourceConversionConfig = ConversionConfig.builder()\n",
+ " .sourceTable(deltaSourceTable)\n",
+ " .targetTables(Arrays.asList(hudiTargetTable))\n",
" .syncMode(SyncMode.INCREMENTAL)\n",
" .build()\n",
- "conversionController.sync(deltaSourcePerTableConfig,
deltaConversionSourceProvider)"
+ "\n",
+ "conversionController.sync(deltaSourceConversionConfig,
deltaConversionSourceProvider)"
]
},
{
@@ -270,8 +327,9 @@
"outputs": [],
"source": [
"// Join the two datasets as Delta tables\n",
- "spark.sql(\"CREATE OR REPLACE TABLE hudi_dimCustomer USING DELTA LOCATION
'/home/data/db/hudi_dimCustomer'\")\n",
- "spark.sql(\"CREATE OR REPLACE TABLE delta_dimGeography USING DELTA
LOCATION '/home/data/db/delta_dimGeography'\")\n",
+
"spark.read.format(\"delta\").load(\"/home/data/db/hudi_dimCustomer\").createOrReplaceTempView(\"hudi_dimCustomer\")\n",
+
"spark.read.format(\"delta\").load(\"/home/data/db/delta_dimGeography\").createOrReplaceTempView(\"delta_dimGeography\")\n",
+ "\n",
"spark.sql(\"SELECT * FROM hudi_dimCustomer INNER JOIN delta_dimGeography
ON hudi_dimCustomer.GeographyKey = delta_dimGeography.GeographyKey\")\n",
" .select(\"FirstName\", \"LastName\", \"CityName\").show(10)"
]
@@ -294,18 +352,22 @@
"outputs": [],
"source": [
"// translate the Delta Lake table to Iceberg\n",
- "val perTableConfig = PerTableConfigImpl.builder()\n",
- " .tableName(deltaTableName)\n",
+ "val deltaToIcebergTargetTable = TargetTable.builder()\n",
+ " .name(deltaTableName)\n",
+ " .formatName(TableFormat.ICEBERG)\n",
+ " .basePath(deltaBasePath)\n",
" .namespace(namespaceArray)\n",
- " .targetTableFormats(Arrays.asList(TableFormat.ICEBERG))\n",
- " .tableBasePath(deltaBasePath)\n",
- " .icebergCatalogConfig(icebergCatalogConfig)\n",
+ " .catalogConfig(icebergCatalogConfig)\n",
+ " .additionalProperties(new Properties())\n",
+ " .build()\n",
+ "\n",
+ "val deltaToIcebergConversionConfig = ConversionConfig.builder()\n",
+ " .sourceTable(deltaSourceTable)\n",
+ " .targetTables(Arrays.asList(deltaToIcebergTargetTable))\n",
" .syncMode(SyncMode.INCREMENTAL)\n",
" .build()\n",
- "val deltaConversionSourceProvider = new
DeltaConversionSourceProvider()\n",
-
"deltaConversionSourceProvider.init(spark.sparkContext.hadoopConfiguration,
Collections.emptyMap())\n",
- "val conversionController = new
ConversionController(spark.sparkContext.hadoopConfiguration)\n",
- "conversionController.sync(perTableConfig, deltaConversionSourceProvider)"
+ "\n",
+ "conversionController.sync(deltaToIcebergConversionConfig,
deltaConversionSourceProvider)"
]
},
{
@@ -335,16 +397,52 @@
"id": "f25f230b-7ff2-4231-add4-4fe1906080ed",
"metadata": {},
"source": [
- ""
+ "<p align=\"center\">\n",
+ " <img src=\"attachment:858ffdfb-a437-43a3-b54f-28a6cf071a1c.png\"
alt=\"Screenshot 2023-11-17 at 10.03.52 AM.png\"/>\n",
+ "</p>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "759ab564",
+ "metadata": {},
+ "source": [
+ "## Validating the synced tables from Trino\n",
+ "Both datasets are now registered in the Hive Metastore as Iceberg tables,
so we can validate the end-to-end\n",
+ "sync results from Trino: row counts for both tables, and the row that was
inserted through Trino earlier and\n",
+ "round-tripped Hudi -> Iceberg -> Delta."
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "dec0e4ef-12da-413c-8bf2-2c859351b1c8",
+ "id": "c6744565",
"metadata": {},
"outputs": [],
- "source": []
+ "source": [
+ "val validationConnection = DriverManager.getConnection(url)\n",
+ "val validationStatement = validationConnection.createStatement()\n",
+ "\n",
+ "def runQuery(query: String): Unit = {\n",
+ " println(s\"> $query\")\n",
+ " val rs = validationStatement.executeQuery(query)\n",
+ " val columnCount = rs.getMetaData.getColumnCount\n",
+ " val header = (1 to
columnCount).map(rs.getMetaData.getColumnName).mkString(\" | \")\n",
+ " println(header)\n",
+ " while (rs.next()) {\n",
+ " println((1 to columnCount).map(rs.getString).mkString(\" | \"))\n",
+ " }\n",
+ " rs.close()\n",
+ " println()\n",
+ "}\n",
+ "\n",
+ "runQuery(\"SELECT COUNT(*) AS row_count FROM
iceberg.demo.hudi_dimCustomer\")\n",
+ "runQuery(\"SELECT COUNT(*) AS row_count FROM
iceberg.demo.delta_dimGeography\")\n",
+ "runQuery(\"SELECT CustomerKey, FirstName, LastName FROM
iceberg.demo.hudi_dimCustomer WHERE _c0 = '18869'\")\n",
+ "\n",
+ "validationStatement.close()\n",
+ "validationConnection.close()"
+ ]
}
],
"metadata": {
@@ -359,7 +457,7 @@
"mimetype": "text/x-scala",
"name": "scala",
"nbconvert_exporter": "script",
- "version": "2.12.18"
+ "version": "2.12.20"
}
},
"nbformat": 4,
diff --git a/demo/trino/catalog/delta.properties b/demo/presto/log.properties
similarity index 71%
copy from demo/trino/catalog/delta.properties
copy to demo/presto/log.properties
index 3da2a722..3ace3ffe 100644
--- a/demo/trino/catalog/delta.properties
+++ b/demo/presto/log.properties
@@ -14,10 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-connector.name=delta_lake
-hive.metastore.uri=thrift://hive-metastore:9083
-hive.metastore-cache-ttl=0s
-hive.metastore-refresh-interval=5s
-hive.metastore-timeout=10s
-delta.enable-non-concurrent-writes=true
-delta.security=ALLOW_ALL
\ No newline at end of file
+## The RMI accept loop behind the JMX agent logs a harmless
+## "accept loop for ServerSocket[...] throws java.net.SocketTimeoutException"
+## warning every time its accept() call times out. It floods the demo logs and
+## reads like a failure, so only surface real errors from that logger.
+sun.rmi.transport.tcp=ERROR
diff --git a/demo/start_demo.sh b/demo/start_demo.sh
index 493c2107..711c83e7 100755
--- a/demo/start_demo.sh
+++ b/demo/start_demo.sh
@@ -15,17 +15,46 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-## Create the require jars for the demo and copy them into a directory we'll
mount in our notebook container
+## Start the demo containers in the background. Builds the XTable jars first
+## (via build_demo.sh) if they are not present. Use stop_demo.sh to stop the
+## demo when you are done.
+
+set -e
CURRENT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
-XTABLE_HOME="$( cd "$(dirname "$CURRENT_DIR")" ; pwd -P )"
-cd $XTABLE_HOME
-./mvnw install -am -pl xtable-core -DskipTests -T 2
-mkdir -p demo/jars
-cp
xtable-hudi-support/xtable-hudi-support-utils/target/xtable-hudi-support-utils-0.5.0-SNAPSHOT.jar
demo/jars
-cp xtable-api/target/xtable-api-0.5.0-SNAPSHOT.jar demo/jars
-cp xtable-core/target/xtable-core_2.12-0.5.0-SNAPSHOT.jar demo/jars
+## Validate the demo jars are built; build them if anything is missing.
+if [ ! -f "${CURRENT_DIR}/jars/versions.properties" ] || ! ls
"${CURRENT_DIR}"/jars/xtable-core_*.jar > /dev/null 2>&1; then
+ echo "XTable demo jars not found. Building the XTable jars first..."
+ "${CURRENT_DIR}/build_demo.sh"
+fi
+
+if [ ! -f "${CURRENT_DIR}/jars/versions.properties" ] || ! ls
"${CURRENT_DIR}"/jars/xtable-core_*.jar > /dev/null 2>&1; then
+ echo "ERROR: the XTable build did not produce the expected demo jars under
demo/jars." >&2
+ echo "Fix the build (see output above) and re-run this script." >&2
+ exit 1
+fi
+
+cd "${CURRENT_DIR}"
+## The first start builds the notebook image (JDK 11 and all notebook
+## dependencies are baked in); later starts reuse the cached image.
+docker-compose up -d --build
+
+echo "Waiting for the Jupyter server to start..."
+JUPYTER_READY=""
+for _ in $(seq 1 150); do
+ if docker logs jupyter 2>&1 | grep -qE "Jupyter Server .* is running at";
then
+ JUPYTER_READY="yes"
+ break
+ fi
+ sleep 2
+done
-cd demo
-docker-compose up
+if [ -n "${JUPYTER_READY}" ]; then
+ echo ""
+ echo "Jupyter is running at: http://localhost:8888/lab";
+ echo "The demo notebooks are under the work/ directory."
+else
+ echo "Jupyter did not report as running yet; check its status with: docker
logs jupyter"
+fi
+echo "Run ./stop_demo.sh when you are done (add --reset-data to restore the
seed datasets)."
diff --git a/demo/start_demo.sh b/demo/stop_demo.sh
similarity index 66%
copy from demo/start_demo.sh
copy to demo/stop_demo.sh
index 493c2107..aa335e6d 100755
--- a/demo/start_demo.sh
+++ b/demo/stop_demo.sh
@@ -15,17 +15,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-## Create the require jars for the demo and copy them into a directory we'll
mount in our notebook container
+## Stop the demo containers started by start_demo.sh. Pass --reset-data to also
+## restore the seed datasets under demo/data to their original state so the
+## demo notebook can be re-run from scratch.
CURRENT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
XTABLE_HOME="$( cd "$(dirname "$CURRENT_DIR")" ; pwd -P )"
-cd $XTABLE_HOME
-./mvnw install -am -pl xtable-core -DskipTests -T 2
-mkdir -p demo/jars
-cp
xtable-hudi-support/xtable-hudi-support-utils/target/xtable-hudi-support-utils-0.5.0-SNAPSHOT.jar
demo/jars
-cp xtable-api/target/xtable-api-0.5.0-SNAPSHOT.jar demo/jars
-cp xtable-core/target/xtable-core_2.12-0.5.0-SNAPSHOT.jar demo/jars
+cd "$CURRENT_DIR"
+docker-compose down
-cd demo
-docker-compose up
+if [ "$1" = "--reset-data" ]; then
+ echo "Resetting demo datasets under demo/data..."
+ cd "$XTABLE_HOME"
+ git checkout -- demo/data
+ git clean -fdxq demo/data
+fi
diff --git a/demo/trino/catalog/delta.properties
b/demo/trino/catalog/delta.properties
index 3da2a722..e05311ff 100644
--- a/demo/trino/catalog/delta.properties
+++ b/demo/trino/catalog/delta.properties
@@ -20,4 +20,5 @@ hive.metastore-cache-ttl=0s
hive.metastore-refresh-interval=5s
hive.metastore-timeout=10s
delta.enable-non-concurrent-writes=true
-delta.security=ALLOW_ALL
\ No newline at end of file
+delta.security=ALLOW_ALL
+delta.register-table-procedure.enabled=true
diff --git a/demo/trino/catalog/iceberg.properties
b/demo/trino/catalog/iceberg.properties
index d59b6861..e114ec60 100644
--- a/demo/trino/catalog/iceberg.properties
+++ b/demo/trino/catalog/iceberg.properties
@@ -19,4 +19,5 @@ hive.metastore.uri=thrift://hive-metastore:9083
hive.metastore-cache-ttl=0s
hive.metastore-refresh-interval=5s
hive.metastore-timeout=10s
-iceberg.catalog.type=hive_metastore
\ No newline at end of file
+iceberg.catalog.type=hive_metastore
+iceberg.register-table-procedure.enabled=true
diff --git a/website/docs/demo/docker.md b/website/docs/demo/docker.md
index 1ee4754c..04100c55 100644
--- a/website/docs/demo/docker.md
+++ b/website/docs/demo/docker.md
@@ -26,6 +26,18 @@ cd demo
./start_demo.sh
```
+The containers run in the background; the script prints the Jupyter notebook
URL once the server is up.
+The XTable jars are built automatically on the first run; they can also be
(re)built explicitly with the
+`build_demo.sh` script.
+
+When you are done with the demo, stop and remove the containers with the
`stop_demo.sh` script.
+Pass `--reset-data` to also restore the seed datasets under `demo/data` to
their original state
+so the demo can be re-run from scratch.
+
+```shell md title="shell"
+./stop_demo.sh --reset-data
+```
+
### Accessing Services
#### Trino
You can access the local Trino container by running `docker exec -it trino
trino`
