yihua commented on code in PR #706: URL: https://github.com/apache/hudi-rs/pull/706#discussion_r3920341825
########## benchmark/filegroup/SKILL.md: ########## @@ -0,0 +1,105 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +--- +name: fg-bench-local +description: Run the hudi-rs file-group-reader performance benchmark locally — build the readers, generate single-FG MOR datasets, and run the containerized resource-tier matrix comparing the Rust hudi-rs reader against the Java gold readers (java-avro pure-JVM, java-spark columnar). Use when the user asks to "run the fg-bench benchmark", "benchmark the hudi-rs reader locally", "compare hudi-rs vs the Java reader", "run the FG reader perf matrix", "regenerate the bench datasets", or similar. Encodes the moved paths, the docker/sudo shim, and the hard-won gotchas (cp exec-bit, java-spark compaction.max.size budget key, cgroup-peak vs true RSS, JDK17 Spark flags) that otherwise silently break a run or make the numbers wrong. +--- + +# fg-bench local performance benchmark + +Goal: measure the **hudi-rs MOR file-group reader** (`HoodieFileGroupReader`) in isolation under controlled CPU/memory tiers, and compare it head-to-head against the two **Java gold readers**. Three readers, one JSON schema, one report: + +| ref | what | engine | in-memory format | +|---|---|---|---| +| `current` | hudi-rs reader (all A1–A6 fixes), `--streaming` | none (Rust) | Arrow columnar | +| `java` | Java `HoodieFileGroupReader` via `HoodieAvroReaderContext` | none (pure JVM) | Avro row (`IndexedRecord`) | +| `java-spark` | production columnar path via embedded `local[N]` Spark | in-process Spark driver | `InternalRow`/`ColumnarBatch` | + +`baseline` (pre-fix Rust binary) is a 4th optional ref if a baseline binary is staged. + +## Paths (verified 2026-06-11; the original `ws3`/`operations` paths are GONE) +- **Repo**: `/home/ubuntu/lin_root/hudi-rs-internal` — branch `lin/mor-on-0607` already contains the A6 work (`benchmark/filegroup/` = the `fg-bench` crate). Review Comment: Done. Rebased on latest main and stripped the internal references: SKILL.md is dropped entirely, and the ENG/milestone/codename references are gone from the bench README and CLI help, the v2 reader test comment, and both fixture docs. The few commit messages that pointed at internal branches and tickets are reworded too. The old strings still exist in intermediate commits of the branch, so this should land as a squash merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
