hudi-agent commented on code in PR #19268: URL: https://github.com/apache/hudi/pull/19268#discussion_r3567878178
########## website/blog/2021-08-23-async-clustering.md: ########## @@ -219,3 +223,13 @@ over yet and future work entails: Please follow this [JIRA](https://issues.apache.org/jira/browse/HUDI-1042) to learn more about active development on this issue. We look forward to contributions from the community. Hope you enjoyed this post. Put your Hudi on and keep streaming! + +## FAQ + +<PostFAQ heading={null} items={[ + {question: 'What is asynchronous clustering in Apache Hudi?', answer: 'Asynchronous clustering runs Hudi\'s clustering table service in the background while regular writers keep ingesting into the table. Hudi\'s multi-writer support provides snapshot isolation between table services, so data can be reorganized for better query performance without compromising ingestion speed.'}, + {question: 'What clustering plan strategies does Hudi provide?', answer: 'Hudi ships three pluggable plan strategies. SparkSizeBasedClusteringPlanStrategy groups small file slices up to a max size per group, SparkRecentDaysClusteringPlanStrategy (the default) clusters small files in the previous N days of partitions, and SparkSelectedPartitionsClusteringPlanStrategy clusters only partitions within a configured begin and end range.'}, Review Comment: π€ This FAQ answer labels `SparkRecentDaysClusteringPlanStrategy` as "(the default)", which faithfully matches the 2021 post body (line 48) but is now out of date β the current clustering docs (`docs/clustering.md`) list `SparkSizeBasedClusteringPlanStrategy` as the default `hoodie.clustering.plan.strategy.class`. Since this answer feeds the FAQPage JSON-LD (intended for AI answer engines) and the post now carries a 2026 `last_update` date, it could lead engines to surface an outdated default. It might help to qualify it the way the indexing-post FAQ does ("At the time of this post, ..."), or to name `SparkSizeBasedClusteringPlanStrategy` as the current default. <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/blog/2019-09-09-ingesting-database-changes.md: ########## @@ -1,6 +1,7 @@ --- title: "Ingesting Database changes via Sqoop/Hudi" excerpt: "Learn how to ingesting changes from a HUDI dataset using Sqoop/Hudi" +description: "Learn how to ingesting changes from a HUDI dataset using Sqoop/Hudi" Review Comment: π€ Now that this `excerpt` is being mirrored into `description` β which Docusaurus surfaces as the page's meta description for search/AI results β it may be worth tightening the wording while you're here: "ingesting changes from a HUDI dataset using Sqoop" reads as if changes are pulled *from* Hudi, whereas the post is about ingesting database changes *into* Hudi via Sqoop. Something like "Learn how to ingest database changes into a Hudi dataset using Sqoop" would make the surfaced snippet clearer. A few other mirrored descriptions may carry over similar phrasing worth a quick pass. <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/blog/2022-07-11-build-open-lakehouse-using-apache-hudi-and-dbt.md: ########## @@ -1,6 +1,7 @@ --- title: "Build Open Lakehouse using Apache Hudi & dbt" excerpt: "How to style blog focused projects on teaching how to build an open Lakehouse using Apache Hudi & dbt" +description: "How to style blog focused projects on teaching how to build an open Lakehouse using Apache Hudi & dbt" Review Comment: π€ The new `description` mirrors an `excerpt` that reads like garbled placeholder copy: "How to style blog focused projects on teaching how to build an open Lakehouse using Apache Hudi & dbt". Since this PR makes `description` the field Docusaurus uses for meta tags and BlogPosting structured data, this sentence will now surface directly in search results and AI answers. It might help to rewrite it into a clean one-liner, e.g. "A hands-on tutorial for building an open lakehouse using Apache Hudi and dbt." <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/blog/2023-11-01-record-level-index.md: ########## @@ -229,3 +241,13 @@ by Hudi metadata tables. [^3] As of now, query engine integration is only available for Spark, with plans to support additional engines in the future. [^4] The query improvement is specific to record-key-matching queries and does not reflect a general reduction in latency by enabling RLI. In the case of the single record-key query, 99.995% of file groups (19999 out of 20000) were pruned during query execution. + +## FAQ + +<PostFAQ heading={null} items={[ + {question: 'What is the Record Level Index in Apache Hudi?', answer: 'The Record Level Index (RLI) is a global index introduced in Hudi 0.14.0 that stores one-to-one mappings between record keys and their file groups in a dedicated partition of Hudi\'s metadata table. It lets writers and readers locate the exact file group for a record key, drastically reducing the number of files that need to be scanned.'}, + {question: 'How do I enable the Record Level Index in Hudi?', answer: 'Set hoodie.metadata.record.index.enable=true and hoodie.index.type=RECORD_INDEX, with the metadata table enabled via hoodie.metadata.enable=true. Since the number of file groups in the RLI partition is fixed at initialization, it is recommended to configure the file group count and size settings appropriately for the expected data volume.'}, Review Comment: π€ This FAQ answer recommends `hoodie.index.type=RECORD_INDEX`, but the current indexing docs (`website/docs/indexes.md`) now mark `RECORD_INDEX` as deprecated, pointing users to `GLOBAL_RECORD_LEVEL_INDEX` (global uniqueness) or `RECORD_LEVEL_INDEX` (partition-level) instead. Since RLI is a global index and this is present-tense "how do I enable" guidance carrying a fresh `last_update` that explicitly directs readers to the current 1.x picture, it might help to use `GLOBAL_RECORD_LEVEL_INDEX` here (or note that `RECORD_INDEX` is now a deprecated alias) so 1.x users don't adopt a deprecated config. <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/blog/2025-11-25-apache-hudi-release-1-1-announcement.md: ########## @@ -1,6 +1,7 @@ --- title: Apache Hudi 1.1 is HereβBuilding the Foundation for the Next Generation of Lakehouse excerpt: '' +description: '' Review Comment: π€ This mirrors an empty `excerpt` into an empty `description`. Given the PR's stated rationale that `description` is what Docusaurus reads for meta tags and structured data, an empty value yields no SEO/AEO benefit on what is a high-traffic release-announcement page. Would it be worth adding a one-line description here (and similarly for the other posts whose excerpt was empty, e.g. the Partition Stats and indexing deep-dive Part 1 posts)? <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/blog/2025-04-02-secondary-index.md: ########## @@ -193,4 +201,14 @@ Indexing has been a core component of Apache Hudi since its inception, enabling Additionally, to ensure that index maintenance does not introduce bottlenecks, Hudiβs *asynchronous indexing* service decouples index updates from ingestion, enabling seamless scaling while keeping indexes timeline-consistent and ACID-compliant. These advancements further solidify Hudiβs role as a high-performance lakehouse platform, making data structures such as secondary indexes more accessible. ---- \ No newline at end of file +--- + +## FAQ + +<PostFAQ heading={null} items={[ + {question: 'What is a secondary index in Apache Hudi?', answer: 'A secondary index, introduced in Hudi 1.0, lets users index columns that are not part of the record key. Hudi stores mappings between secondary key values and record keys in its metadata table, so queries filtering on non-primary-key fields can prune files via data skipping instead of scanning the full table.'}, + {question: 'How do I create a secondary index in Hudi?', answer: 'On a table with the record index enabled, run a SQL statement such as CREATE INDEX idx_city ON hudi_table(city). Secondary indexes can also be configured through the Spark DataSource API using hoodie.metadata.index.secondary.enable and hoodie.datasource.write.secondarykey.column.'}, + {question: 'How much does a secondary index improve query performance?', answer: 'In a TPCDS 1TB benchmark with an index on the web_sales table, the same join query ran about 33% faster on the first run and 58% faster on the second, while the data scanned dropped by roughly 90%, from 67GB across 5000 files to 7GB across 521 files.'}, + {question: 'Which query engines support Hudi secondary indexes?', answer: 'In Hudi 1.0, secondary indexes are supported in Apache Spark, with support for Flink, Presto, and Trino planned for Hudi 1.1. Reduced data scans particularly benefit cloud query engines like AWS Athena that price by data scanned.'}, Review Comment: π€ This newly-added FAQ states that Flink, Presto, and Trino support is "planned for Hudi 1.1." With Hudi 1.1 (and now 1.2) released, that forward-looking phrasing may read as stale to a 2026 reader, especially since the post now carries a recent `last_update`. Could you confirm whether that engine support actually landed and update the wording accordingly? If the intent is to preserve the historical 1.0 framing, it's reasonable to leave as-is with the "In Hudi 1.0" qualifier. <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/src/pages/faq/general.md: ########## @@ -7,10 +7,28 @@ keywords: [hudi, writing, reading] ### When is Hudi useful for me or my organization? -If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. +If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. Hudi remains the de facto lakehouse format for fast incremental writes and reads, and it ships with automated table maintenance built in, so tables stay optimized without external orchestration. As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker. +### What makes Hudi different from other lakehouse formats? + +Hudi offers a set of core capabilities today that other lakehouse formats do not. The [21 unique differentiators](/blog/2025/03/05/hudi-21-unique-differentiators) post covers the technical crux in depth; the highlights are: + +* **_Multi-modal indexing:_** Hudi maintains a range of [indexes](/docs/indexes) β record-level indexes, bloom filters, bucket indexes and more β that speed up upserts and deletes on the write side, plus read-side secondary indexes (including expression indexes on columns) that prune queries, much like a relational database. +* **_Non-blocking concurrency control:_** Hudi's MVCC-based [concurrency control](/docs/concurrency_control#non-blocking-concurrency-control) lets multiple writers and table services modify a table concurrently without failing or blocking each other, avoiding wasted compute from retries and livelocks. +* **_Async compaction and built-in table services:_** compaction, clustering, cleaning, file sizing, indexing and archival are scheduled and executed automatically alongside writes β no external orchestration or manual maintenance commands. Hudi is the only lakehouse project that can rapidly ingest data while handling small-file compaction without blocking those writes. This kind of table maintenance is something you typically pay a vendor for; in Hudi it is open source and built in. Review Comment: π€ The assertion that "Hudi is the only lakehouse project that can rapidly ingest data while handling small-file compaction without blocking those writes" is a strong superlative that other projects could dispute, since several formats now offer async or inline compaction. For an FAQ that AI answer engines may quote verbatim, it might help to either substantiate what is uniquely Hudi here (e.g., NBCC plus async table services running concurrently with writers, without failing them) or soften the "only" phrasing so the claim stays defensible. A committer/PMC member may want to weigh in on the exact framing given it compares against other Apache projects. <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/src/pages/faq/general.md: ########## @@ -7,10 +7,28 @@ keywords: [hudi, writing, reading] ### When is Hudi useful for me or my organization? -If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. +If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. Hudi remains the de facto lakehouse format for fast incremental writes and reads, and it ships with automated table maintenance built in, so tables stay optimized without external orchestration. As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker. +### What makes Hudi different from other lakehouse formats? + +Hudi offers a set of core capabilities today that other lakehouse formats do not. The [21 unique differentiators](/blog/2025/03/05/hudi-21-unique-differentiators) post covers the technical crux in depth; the highlights are: + +* **_Multi-modal indexing:_** Hudi maintains a range of [indexes](/docs/indexes) β record-level indexes, bloom filters, bucket indexes and more β that speed up upserts and deletes on the write side, plus read-side secondary indexes (including expression indexes on columns) that prune queries, much like a relational database. +* **_Non-blocking concurrency control:_** Hudi's MVCC-based [concurrency control](/docs/concurrency_control#non-blocking-concurrency-control) lets multiple writers and table services modify a table concurrently without failing or blocking each other, avoiding wasted compute from retries and livelocks. +* **_Async compaction and built-in table services:_** compaction, clustering, cleaning, file sizing, indexing and archival are scheduled and executed automatically alongside writes β no external orchestration or manual maintenance commands. Hudi is the only lakehouse project that can rapidly ingest data while handling small-file compaction without blocking those writes. This kind of table maintenance is something you typically pay a vendor for; in Hudi it is open source and built in. +* **_Ingestion utilities:_** production-ready [ingestion tools](/docs/hoodie_streaming_ingestion) like Hudi Streamer and the Flink writer build lakehouse tables from Kafka, Pulsar, S3/GCS and popular CDC formats (Debezium, AWS DMS, Mongo) with a single command. +* **_Blob/unstructured data support:_** starting with Hudi 1.2, tables can manage blob and unstructured data alongside structured records, extending the lakehouse beyond tabular workloads. + +Combined with a storage format that balances write speed and query performance, these capabilities make Hudi the leader in incremental write performance and the de facto format for fast incremental writes and reads. + +### How does Hudi relate to Apache Iceberg? Are Hudi tables compatible with Iceberg? + +The two projects were engineered around different workloads. Iceberg's design centers on the traditional batch, scan-oriented workloads that Apache Hive served β large periodic rewrites and full-table scans. Hudi was engineered for fast-moving, mutable data: streaming ingestion, CDC, record-level upserts and deletes, and incremental pipelines that process only what changed. Choosing between them is a question of workload fit, not either/or on data access. + +That is because Hudi tables (copy-on-write) are fully format-compatible with Iceberg readers. [Apache XTable](/docs/syncing_xtable) (incubating) translates Hudi table metadata into Iceberg metadata in place β no data is copied or rewritten β so a single copy of data on cloud storage is readable as both Hudi and Iceberg. You can ingest and manage tables with Hudi's write-side strengths while any Iceberg-only engine, BI tool or catalog queries the same data. Review Comment: π€ This section could clarify that the Iceberg compatibility is achieved through Apache XTable's metadata translation rather than native readability. As written, "Hudi tables (copy-on-write) are fully format-compatible with Iceberg readers" reads as out-of-the-box interoperability, but an Iceberg engine cannot query a Hudi table until XTable has generated the Iceberg metadata β it is the shared Parquet data files that are compatible, with XTable bridging the metadata. Leading with the XTable mechanism (or softening "fully format-compatible") would remove the tension with the very next sentence. Good call scoping this to copy-on-write; it might help to note explicitly that MOR tables need compaction/merge first, since unmerged log files would not be visible to Iceberg readers. <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> ########## website/docs/migration_guide.md: ########## @@ -97,6 +97,15 @@ hudi->bootstrap run --srcPath /tmp/source_table --targetPath /tmp/hoodie/bootstr ``` Unlike Hudi Streamer, FULL_RECORD or METADATA_ONLY is set with --selectorClass, see details with help "bootstrap run". +### Migrating from Delta Lake or Apache Iceberg + +Tables already managed by Delta Lake or Apache Iceberg store their data as Parquet files, so they can be migrated to Review Comment: π€ The opening sentence here appears to assume every source table is Parquet-backed. That always holds for Delta Lake, but Apache Iceberg tables can also store data as ORC or Avro. Because XTable's no-rewrite path works by translating metadata over the existing data files, it only applies when those files are Parquet β an ORC/Avro Iceberg table would fall back to the full Spark rewrite mentioned at the end. It might help to qualify the claim (e.g., "store their data as Parquet files in the common case") and note that non-Parquet sources need the rewrite path. <sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
