vinishjail97 opened a new pull request, #935: URL: https://github.com/apache/incubator-xtable/pull/935
> Resolves #934. **Stacked on #835.** Only the last commit belongs to this PR; the earlier commits are the Hudi table version 9 work and will disappear once #835 merges. The PR also depends on apache/hudi#19869 (record index and secondary index for external parquet files), so it stays a draft until that change ships in a Hudi release and `hudi.version` is re-pinned. ## What is the purpose of the pull request Lets the Hudi conversion target build a Hudi **record index** and **secondary index** for the files it registers, and adds a `HudiBackedIcebergSecondaryIndex` that uses those indexes to resolve values of an Iceberg column to the data file, row position and partition that hold them. Engines can use the lookup to locate the rows touched by a merge or delete without joining against the whole table. The files XTable registers carry no Hudi record keys, so Hudi keys each row by `<file path relative to the table>_<row position>` (apache/hudi#19869). A secondary index lookup therefore returns the file and the row position directly, which is what the index exposes as `IndexLookupResult` (mirroring Iceberg's `_file`, `_pos` and `_partition` metadata columns). ## Brief change log **Hudi target configuration** (`HudiTargetConfig`) - `xtable.hudi.target.secondary.index.column`: column to build the secondary index on. Enables the global record index as well, since the secondary index resolves through it. Requires `xtable.hudi.target.table_version=9`. - `xtable.hudi.target.metadata.record.index.min.filegroup.count` / `...max.filegroup.count`: file group bounds of the record index (must be set together). - `xtable.hudi.target.metadata.index.secondary.parallelism`: parallelism of the secondary index generation. - `xtable.hudi.target.execution.engine`: `java` (default) or `spark`. Building the record and secondary indexes reads every registered file, so the Spark engine is the practical choice for those. **Execution engines** (`org.apache.xtable.hudi.engine`) - `HudiExecutionEngineProvider` abstracts the engine context, `HoodieTable` creation and the commit. `JavaExecutionEngineProvider` keeps today's behaviour; `SparkExecutionEngineProvider` commits through `SparkRDDWriteClient` on the active Spark session. - `HudiConversionTarget` takes the provider and the parsed `HudiTargetConfig`, enables the indexes on the metadata table config when a secondary index column is set, and sets `_hoodie.writes.fileid.encoding=1` because the registered file names are not UUID based Hudi file ids. - `hudi-spark-client` is added as a `provided` dependency of `xtable-core`, with its Lance and Vortex artifacts excluded because they are compiled against Spark 3.5; the Spark engine expects the Hudi Spark bundle on the classpath of the Spark application. **Index API** (`org.apache.xtable.index`) - `Index<T>` with `doesIndexExist`, `syncIndex` and `lookup`. - `HudiBackedIcebergSecondaryIndex` syncs the Iceberg table to Hudi (table version 9, Spark engine, secondary index on the requested column) under `<data location>/.hoodie` and looks keys up with `HoodieBackedTableMetadata#readSecondaryIndexDataTableRecordKeysWithKeys`. - `IcebergPartitionConverter` turns the partition path of a matched file into the Spark `InternalRow` Iceberg exposes as `_partition`. ## Verify this pull request This change added tests and can be verified as follows: - `ITHudiBackedIcebergSecondaryIndex`: builds the index for a partitioned and an unpartitioned Iceberg table, checks that every lookup resolves to the same file and row position as Iceberg's `_file`/`_pos` metadata columns and to the right partition value, that unknown keys return nothing, and that a second sync picks up newly added files. - `TestHudiTargetConfig`: parsing and validation of the new properties. - `TestHudiConversionTarget`: updated for the engine provider. - `TestIcebergPartitionConverter`: type conversion of partition values. Run locally against a Hudi 1.3.0-SNAPSHOT built from apache/hudi#19869 (`-Dspark3.4 -Dscala-2.12`): xtable-core unit tests and `ITHudiBackedIcebergSecondaryIndex`, `ITHudiConversionTarget`, `ITHudiConversionSource` pass; `ITConversionController` result to be posted below. ## Limitations / follow-ups - The index is populated from the commits XTable writes. Enabling a secondary index on a Hudi target that already exists is not supported yet, because Hudi's index bootstrap reads the record key meta field, which the registered files do not have. - Iceberg tables with several partition specs are resolved with the current spec only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
