vinishjail97 opened a new pull request, #992:
URL: https://github.com/apache/incubator-xtable/pull/992
## What
Spark SQL extensions for the secondary index from #935, so Iceberg users on
Spark can create the index with SQL and have point lookups use it without
changing their queries. The extensions ship in `xtable-spark-runtime`, so no
other XTable jar is needed.
Part of #887.
## User experience
**Setup.** Put `xtable-spark-runtime`, `hudi-spark3.x-bundle` and
`iceberg-spark-runtime-3.x` in one folder on the driver and executor classpath,
and add the extensions after Iceberg's:
```properties
spark.driver.extraClassPath=/opt/jars/*
spark.executor.extraClassPath=/opt/jars/*
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.apache.xtable.spark.extensions.XTableSparkSessionExtensions
spark.serializer=org.apache.spark.serializer.KryoSerializer
```
Any Iceberg catalog Spark is configured with works (Glue, Hive, REST,
Hadoop).
**Spark SQL users** manage the index with SQL, and queries stay as they are:
```sql
CREATE INDEX email_idx ON glue.db.customer USING xtable (c_email_address);
-- reads only the files that hold these keys
SELECT * FROM glue.db.customer WHERE c_email_address IN ('[email protected]',
'[email protected]');
REFRESH INDEX email_idx ON glue.db.customer; -- after new snapshots
DROP INDEX email_idx ON glue.db.customer;
```
**Spark jobs** that write the table refresh the index as the last step, and
readers use it from then on:
```scala
df.writeTo("glue.db.customer").append()
spark.sql("REFRESH INDEX email_idx ON glue.db.customer")
```
A `RunSync` Hudi target with `xtable.hudi.target.secondary.index.column` set
to the same column keeps the index current as well.
Results never depend on the index. Spark reads the table as usual when the
index is behind the current snapshot, for time travel and branch reads, and for
scans that Iceberg's own pruning already makes small.
## How
- New source module `xtable-spark-extensions` (Scala Catalyst code), bundled
by `xtable-spark-runtime` without relocation, the same split as Iceberg's
`iceberg-spark-extensions` and `iceberg-spark-runtime`.
- `CREATE` and `REFRESH` run a regular sync to a Hudi target at the table's
data path with `xtable.hudi.target.secondary.index.column`, the way `RunSync`
builds the index. `DROP` deletes the Hudi metadata and keeps the data files.
The definition is kept in the table properties (`xtable.index.<name>.column`).
- The pruning rule handles `col = literal`, `IN` and `InSet` on the indexed
column. It looks the keys up with `HudiBackedSecondaryIndexReader`, checks
`getLastSyncedSourceIdentifier()` against the current snapshot, and stages only
the matching files as the scan's task set (`scan-task-set-id`), since Iceberg
does not prune on `_file`. The filter stays in the plan.
- The root pom manages `scala-maven-plugin`, and Spotless checks Scala with
scalafmt.
No changes to `xtable-core` or `xtable-api`.
## Limitations
One index per table on one top level `string`, `int` or `bigint` column.
Joins, `MERGE`, `UPDATE` and `DELETE` do not use the index. Delta tables are
out of scope.
## Testing
- `ITXTableIndexExtensions` (partitioned and unpartitioned): every query is
compared with the same query without the rule, and the staged files with the
files that hold matching rows. Also passes when built against Spark 3.5.2.
- `ITXTableSparkRuntimeBundle` checks that the bundle ships the extensions
unrelocated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
