yihua opened a new pull request, #718:
URL: https://github.com/apache/hudi-rs/pull/718
## Description
The TPC-H harness could only be driven from a GCP VM against GCS. This adds
the AWS counterpart, so either cloud is a documented path: `bootstrap.sh` and
`sync.sh` under `benchmark/tpch/infra/aws` mirroring the GCP pair, an `s3://`
filesystem mapping for the Spark leg (Hadoop 3 dropped that scheme, so it needs
routing to S3A), and a README section alongside the GCP one.
Two details that are easy to get wrong on AWS and are handled here:
`object_store` takes no region from instance metadata and falls back to
`us-east-1`, so bootstrap persists `AWS_REGION`; and the instance store is
wiped on stop/start, so the mount step sits outside the script's sentinel guard
while the package installs stay guarded.
Also creates `$SPARK_HOME/conf` before copying configs into it. A
pip-installed PySpark ships no `conf` directory, so `create-tables` failed
there, which is how the missing directory surfaced.
## How are the changes test-covered
- [ ] N/A
- [ ] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Ran the full SF1 pipeline (`generate`, `create-tables`, both engines,
`compare`) on macOS with Corretto 17 and a pip-installed PySpark 3.5.8, which
is the configuration that exposed the missing `conf` directory:
datafusion+hudi-rs 1840.9 ms total / 74.9 ms geomean vs spark+hudi 32708.3 ms /
1254.5 ms across the 22 queries. Re-ran both engines afterwards starting from a
deleted `$SPARK_HOME/conf` to confirm the fix, and checked both new scripts
with `bash -n` and shellcheck.
The AWS scripts themselves are not exercised by this run; they are scheduled
to be used for an SF100 comparison on EC2 next.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
