rangareddy commented on issue #9612:
URL: https://github.com/apache/hudi/issues/9612#issuecomment-4841541708
## Step-by-Step Remediation Plan
Ordered from lowest-risk/no-upgrade to the durable fix. Steps 1–4 are doable
on 0.11 today; Step 5 is the real resolution.
**Step 1 — Confirm the diagnosis (read-path instrumentation)**
1.1 Re-run the same `SELECT *` and capture the S3 access-log aggregation you
already have, but tag it by query so you can measure deltas after each step.
1.2 Confirm the `5×` multiplier is per-query FS-view rebuilds: check the
Spark driver logs for repeated "Loading Active commit timeline" / FS-view
construction lines within one query.
1.3 Record a baseline: HEAD count on `.hoodie/`, GET count on
`.replacecommit`, and HEAD/GET on `.hoodie_partition_metadata`. These are your
three KPIs.
**Step 2 — Make the metadata table serve listing on the READ path**
2.1 Confirm MDT is healthy: `ls .hoodie/metadata/files/` exists and has
recent deltacommits.
2.2 Set on the **reader/query** session (not just the writer):
```
hoodie.metadata.enable=true
hoodie.metadata.index.column.stats.enable=false # unless you need it
```
2.3 Re-run the query and re-measure. Expectation: the per-partition
`.hoodie_partition_metadata` HEAD/GET and the parquet existence HEADs drop,
because file listing now comes from the MDT `files` partition instead of S3
directory probing.
2.4 If counts don't drop, the read engine isn't routing through MDT — verify
the query path (Spark datasource vs Athena/Presto) actually honors
`hoodie.metadata.enable` on 0.11.
**Step 3 — Shrink the active timeline (cut the `.replacecommit` GET
fan-out)**
3.1 Inspect how many active instants exist: `ls .hoodie/*.replacecommit | wc
-l`. Each one is GET-read on every FS-view build.
3.2 Tighten archival + cleaning on the writer:
```
hoodie.keep.min.commits=20
hoodie.keep.max.commits=30
hoodie.cleaner.commits.retained=10
```
(adjust only if you have a real incremental-query lookback requirement).
3.3 Run one write/commit so archival moves old `replacecommit`s out of the
active timeline into `.hoodie/archived/`.
3.4 Re-run the query and re-measure: the `.replacecommit` GET count should
fall roughly in proportion to the reduced active-instant count.
**Step 4 — Build the FS view once per query (stop the rebuilds)**
4.1 Enable the embedded timeline server for the query/job session so the FS
view is constructed once and cached rather than re-listed per task:
```
hoodie.embed.timeline.server=true
hoodie.filesystem.view.type=MEMORY
```
4.2 Re-run and re-measure the `.hoodie/` HEAD count — this is the step that
targets the `5×` (and the production 700k) directly.
4.3 If you run in a service mesh / restricted network, ensure the ETS port
is reachable from executors to the driver (otherwise it silently falls back to
per-task listing).
**Step 5 — Upgrade off 0.11 (the durable fix)**
5.1 This is what actually removes the redundant `.hoodie` reads and the dead
bootstrap-index HEAD probe: PRs #7404 / #7436 plus the broader
timeline/FS-view/listing rework landed after 0.11.
5.2 De-risk the upgrade in stages rather than jumping to the latest:
- Stand up a **0.12.3** (or 0.13.x) copy of one affected table in a
non-prod path.
- Run the Step 1 instrumentation and compare the three KPIs against your
0.11 baseline.
- Validate read/write correctness and your downstream queries before
promoting.
5.3 Confirm the table is not flagged bootstrapped (check `hoodie.properties`
for any `hoodie.bootstrap.*` keys) so the `.aux/.bootstrap/.partitions` HEAD
probe is eliminated — it should disappear on the upgraded version for a native
table.
5.4 Promote table-by-table, not all at once, given your stated
production-stability constraints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
