codope opened a new pull request, #10153:
URL: https://github.com/apache/hudi/pull/10153
### Change Logs
With files and functional index, it should be possible to read the table
from spark-sql attached to an external catalog, say remote Hive Metastore, even
when the partition metadata is not syned to the catalog.
- Check whether partitions are passed to `HoodieCatalog`. If not, then use
the metadata file system view to get the partitions.
- Add a test for querying a table without syncing partition metadata.
NOTE:
1. The change is in `HoodieCatalog` which is dependent on Spark 3.2. So,
this feature is not available for Spark versions < 3.2.0.
2. Currently, the PR assumes that the partition paths will be hive-style
partitioned.
### Impact
Query a table from spark-sql that is syned to catalog without the partition
metadata. This means we can make meta sync faster and do away with heavy
partition metadata.
### Risk level (write none, low medium or high below)
medium
### Documentation Update
_Describe any necessary documentation update if there is any new feature,
config, or user-facing change_
- _The config description must be updated if new configs are added or the
default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website.
Please create a Jira ticket, attach the
ticket number here and follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to
make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
