vinothchandar commented on code in PR #19268: URL: https://github.com/apache/hudi/pull/19268#discussion_r3567920463
########## website/blog/2021-08-23-async-clustering.md: ########## @@ -219,3 +223,13 @@ over yet and future work entails: Please follow this [JIRA](https://issues.apache.org/jira/browse/HUDI-1042) to learn more about active development on this issue. We look forward to contributions from the community. Hope you enjoyed this post. Put your Hudi on and keep streaming! + +## FAQ + +<PostFAQ heading={null} items={[ + {question: 'What is asynchronous clustering in Apache Hudi?', answer: 'Asynchronous clustering runs Hudi\'s clustering table service in the background while regular writers keep ingesting into the table. Hudi\'s multi-writer support provides snapshot isolation between table services, so data can be reorganized for better query performance without compromising ingestion speed.'}, + {question: 'What clustering plan strategies does Hudi provide?', answer: 'Hudi ships three pluggable plan strategies. SparkSizeBasedClusteringPlanStrategy groups small file slices up to a max size per group, SparkRecentDaysClusteringPlanStrategy (the default) clusters small files in the previous N days of partitions, and SparkSelectedPartitionsClusteringPlanStrategy clusters only partitions within a configured begin and end range.'}, Review Comment: Fixed — the FAQ now notes SparkRecentDaysClusteringPlanStrategy was the default when the post was written and that current releases default to SparkSizeBasedClusteringPlanStrategy. ########## website/blog/2019-09-09-ingesting-database-changes.md: ########## @@ -1,6 +1,7 @@ --- title: "Ingesting Database changes via Sqoop/Hudi" excerpt: "Learn how to ingesting changes from a HUDI dataset using Sqoop/Hudi" +description: "Learn how to ingesting changes from a HUDI dataset using Sqoop/Hudi" Review Comment: Fixed — description now reads "Learn how to ingest database changes into a Hudi dataset using Sqoop and Hudi". ########## website/blog/2023-11-01-record-level-index.md: ########## @@ -229,3 +241,13 @@ by Hudi metadata tables. [^3] As of now, query engine integration is only available for Spark, with plans to support additional engines in the future. [^4] The query improvement is specific to record-key-matching queries and does not reflect a general reduction in latency by enabling RLI. In the case of the single record-key query, 99.995% of file groups (19999 out of 20000) were pruned during query execution. + +## FAQ + +<PostFAQ heading={null} items={[ + {question: 'What is the Record Level Index in Apache Hudi?', answer: 'The Record Level Index (RLI) is a global index introduced in Hudi 0.14.0 that stores one-to-one mappings between record keys and their file groups in a dedicated partition of Hudi\'s metadata table. It lets writers and readers locate the exact file group for a record key, drastically reducing the number of files that need to be scanned.'}, + {question: 'How do I enable the Record Level Index in Hudi?', answer: 'Set hoodie.metadata.record.index.enable=true and hoodie.index.type=RECORD_INDEX, with the metadata table enabled via hoodie.metadata.enable=true. Since the number of file groups in the RLI partition is fixed at initialization, it is recommended to configure the file group count and size settings appropriately for the expected data volume.'}, Review Comment: Fixed — the FAQ now recommends hoodie.metadata.global.record.level.index.enable=true and hoodie.index.type=GLOBAL_RECORD_LEVEL_INDEX, noting the pre-1.1 names are deprecated (per docs/indexes.md). ########## website/blog/2022-07-11-build-open-lakehouse-using-apache-hudi-and-dbt.md: ########## @@ -1,6 +1,7 @@ --- title: "Build Open Lakehouse using Apache Hudi & dbt" excerpt: "How to style blog focused projects on teaching how to build an open Lakehouse using Apache Hudi & dbt" +description: "How to style blog focused projects on teaching how to build an open Lakehouse using Apache Hudi & dbt" Review Comment: Fixed — rewritten to "A hands-on tutorial for building an open lakehouse using Apache Hudi and dbt." (both excerpt and description). ########## website/blog/2025-04-02-secondary-index.md: ########## @@ -193,4 +201,14 @@ Indexing has been a core component of Apache Hudi since its inception, enabling Additionally, to ensure that index maintenance does not introduce bottlenecks, Hudi’s *asynchronous indexing* service decouples index updates from ingestion, enabling seamless scaling while keeping indexes timeline-consistent and ACID-compliant. These advancements further solidify Hudi’s role as a high-performance lakehouse platform, making data structures such as secondary indexes more accessible. ---- \ No newline at end of file +--- + +## FAQ + +<PostFAQ heading={null} items={[ + {question: 'What is a secondary index in Apache Hudi?', answer: 'A secondary index, introduced in Hudi 1.0, lets users index columns that are not part of the record key. Hudi stores mappings between secondary key values and record keys in its metadata table, so queries filtering on non-primary-key fields can prune files via data skipping instead of scanning the full table.'}, + {question: 'How do I create a secondary index in Hudi?', answer: 'On a table with the record index enabled, run a SQL statement such as CREATE INDEX idx_city ON hudi_table(city). Secondary indexes can also be configured through the Spark DataSource API using hoodie.metadata.index.secondary.enable and hoodie.datasource.write.secondarykey.column.'}, + {question: 'How much does a secondary index improve query performance?', answer: 'In a TPCDS 1TB benchmark with an index on the web_sales table, the same join query ran about 33% faster on the first run and 58% faster on the second, while the data scanned dropped by roughly 90%, from 67GB across 5000 files to 7GB across 521 files.'}, + {question: 'Which query engines support Hudi secondary indexes?', answer: 'In Hudi 1.0, secondary indexes are supported in Apache Spark, with support for Flink, Presto, and Trino planned for Hudi 1.1. Reduced data scans particularly benefit cloud query engines like AWS Athena that price by data scanned.'}, Review Comment: Checked the current docs: querying_data.md states Trino dropped Hudi metadata-table reads in Trino 419 (apache/hudi#16286), so secondary-index pruning isn't available through Trino today. Updated the FAQ to say SI-based pruning is available through Spark, and that engines like Trino query Hudi tables without applying secondary-index pruning. If there's newer Trino work that restores metadata-table reads, happy to update both this FAQ and the docs. ########## website/blog/2025-11-25-apache-hudi-release-1-1-announcement.md: ########## @@ -1,6 +1,7 @@ --- title: Apache Hudi 1.1 is Here—Building the Foundation for the Next Generation of Lakehouse excerpt: '' +description: '' Review Comment: Fixed — filled descriptions for all 7 posts that had empty ones (1.1 release announcement, Partition Stats, indexing deep-dive Part 1, auto-gen keys, 2025 year in review, Funding Circle, Upstox). ########## website/src/pages/faq/general.md: ########## @@ -7,10 +7,28 @@ keywords: [hudi, writing, reading] ### When is Hudi useful for me or my organization? -If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. +If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. Hudi remains the de facto lakehouse format for fast incremental writes and reads, and it ships with automated table maintenance built in, so tables stay optimized without external orchestration. As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker. +### What makes Hudi different from other lakehouse formats? + +Hudi offers a set of core capabilities today that other lakehouse formats do not. The [21 unique differentiators](/blog/2025/03/05/hudi-21-unique-differentiators) post covers the technical crux in depth; the highlights are: + +* **_Multi-modal indexing:_** Hudi maintains a range of [indexes](/docs/indexes) — record-level indexes, bloom filters, bucket indexes and more — that speed up upserts and deletes on the write side, plus read-side secondary indexes (including expression indexes on columns) that prune queries, much like a relational database. +* **_Non-blocking concurrency control:_** Hudi's MVCC-based [concurrency control](/docs/concurrency_control#non-blocking-concurrency-control) lets multiple writers and table services modify a table concurrently without failing or blocking each other, avoiding wasted compute from retries and livelocks. +* **_Async compaction and built-in table services:_** compaction, clustering, cleaning, file sizing, indexing and archival are scheduled and executed automatically alongside writes — no external orchestration or manual maintenance commands. Hudi is the only lakehouse project that can rapidly ingest data while handling small-file compaction without blocking those writes. This kind of table maintenance is something you typically pay a vendor for; in Hudi it is open source and built in. +* **_Ingestion utilities:_** production-ready [ingestion tools](/docs/hoodie_streaming_ingestion) like Hudi Streamer and the Flink writer build lakehouse tables from Kafka, Pulsar, S3/GCS and popular CDC formats (Debezium, AWS DMS, Mongo) with a single command. +* **_Blob/unstructured data support:_** starting with Hudi 1.2, tables can manage blob and unstructured data alongside structured records, extending the lakehouse beyond tabular workloads. + +Combined with a storage format that balances write speed and query performance, these capabilities make Hudi the leader in incremental write performance and the de facto format for fast incremental writes and reads. + +### How does Hudi relate to Apache Iceberg? Are Hudi tables compatible with Iceberg? + +The two projects were engineered around different workloads. Iceberg's design centers on the traditional batch, scan-oriented workloads that Apache Hive served — large periodic rewrites and full-table scans. Hudi was engineered for fast-moving, mutable data: streaming ingestion, CDC, record-level upserts and deletes, and incremental pipelines that process only what changed. Choosing between them is a question of workload fit, not either/or on data access. + +That is because Hudi tables (copy-on-write) are fully format-compatible with Iceberg readers. [Apache XTable](/docs/syncing_xtable) (incubating) translates Hudi table metadata into Iceberg metadata in place — no data is copied or rewritten — so a single copy of data on cloud storage is readable as both Hudi and Iceberg. You can ingest and manage tables with Hudi's write-side strengths while any Iceberg-only engine, BI tool or catalog queries the same data. Review Comment: Reworked — the answer now leads with the mechanism (shared Parquet files + XTable metadata translation) before the compatibility claim, and notes merge-on-read tables expose their compacted read-optimized view to Iceberg readers. ########## website/docs/migration_guide.md: ########## @@ -97,6 +97,15 @@ hudi->bootstrap run --srcPath /tmp/source_table --targetPath /tmp/hoodie/bootstr ``` Unlike Hudi Streamer, FULL_RECORD or METADATA_ONLY is set with --selectorClass, see details with help "bootstrap run". +### Migrating from Delta Lake or Apache Iceberg + +Tables already managed by Delta Lake or Apache Iceberg store their data as Parquet files, so they can be migrated to Review Comment: Keeping as-is per the discussion — Parquet is the dominant case, and the paragraph already ends by pointing non-fitting tables at the Spark rewrite path. ########## website/src/pages/faq/general.md: ########## @@ -7,10 +7,28 @@ keywords: [hudi, writing, reading] ### When is Hudi useful for me or my organization? -If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. +If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](/docs/hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data. Hudi remains the de facto lakehouse format for fast incremental writes and reads, and it ships with automated table maintenance built in, so tables stay optimized without external orchestration. As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker. +### What makes Hudi different from other lakehouse formats? + +Hudi offers a set of core capabilities today that other lakehouse formats do not. The [21 unique differentiators](/blog/2025/03/05/hudi-21-unique-differentiators) post covers the technical crux in depth; the highlights are: + +* **_Multi-modal indexing:_** Hudi maintains a range of [indexes](/docs/indexes) — record-level indexes, bloom filters, bucket indexes and more — that speed up upserts and deletes on the write side, plus read-side secondary indexes (including expression indexes on columns) that prune queries, much like a relational database. +* **_Non-blocking concurrency control:_** Hudi's MVCC-based [concurrency control](/docs/concurrency_control#non-blocking-concurrency-control) lets multiple writers and table services modify a table concurrently without failing or blocking each other, avoiding wasted compute from retries and livelocks. +* **_Async compaction and built-in table services:_** compaction, clustering, cleaning, file sizing, indexing and archival are scheduled and executed automatically alongside writes — no external orchestration or manual maintenance commands. Hudi is the only lakehouse project that can rapidly ingest data while handling small-file compaction without blocking those writes. This kind of table maintenance is something you typically pay a vendor for; in Hudi it is open source and built in. Review Comment: Keeping the phrasing as-is per the discussion — the claim is grounded in NBCC plus async table services running concurrently with writers without failing them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
