prashantwason opened a new pull request #2565:
URL: https://github.com/apache/hudi/pull/2565
## What is the purpose of the pull request
During the bootstrap of the Metadata Table, all the directories which
contain the partition metadata directory are assumed to be partitions and are
added to the metadata table.
In our HDFS clusters, we have directories like .backup, .temp which are used
by various teams for non-hoodie purposes (e.g. .backup may be keeping a
snapshot of the dataset). During bootstrap, Metadata Table ends up containing
all those paths also as partitions.
In this patch, I would like to introduce a configuration for
HoodieMetadataConfig to filter out some directories based on a regular
expression string.
## Brief change log
Added a config to HoodieMetadataConfig.
Added a unit test.
## Verify this pull request
This change added tests and can be verified as follows:
Extended the unit test TestHoodieBackedMetadata#testOnlyValidPartitionsAdded
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
