akashchamp opened a new pull request, #20220:
URL: https://github.com/apache/hudi/pull/20220
### Describe the issue this Pull Request addresses
Closes #19879
`export instants` in hudi-cli had three bugs:
1. It globbed the bare archive path. Without a wildcard the glob returns the
directory itself, and the command then opened that directory as a log file. On
a v8+ table this is `.hoodie/timeline/history`, and on an older table it is
`.hoodie/archived`. Any export that walks the archive, which includes the
default ascending order, failed with `HoodieIOException: Exception creating
input stream from file`. The legacy log reader also could not read the LSM
history even with the right path.
2. `--limit` was ignored for active instants.
3. `restore` is in the default `--actions` list, but a completed restore
instant made the export throw `Unknown type of action restore`.
### Summary and Changelog
`export instants` now works on v8+ tables in either order, honours
`--limit`, and exports restore instants.
- On a table of version 8 or above, archived instants are read through the
archived timeline (`metaClient.getArchivedTimeline("", false)`), the same way
#18816 reworked `ArchivedCommitsCommand`. Only the time range being exported is
loaded. Each payload is written as JSON named `<instant>.<action>`, which
matches what the legacy branch writes.
- Older tables keep the legacy log reader, with the `.commits_.archive*`
glob put back. The legacy reader now counts exported instants instead of log
blocks. It also skips the requested and inflight entries, so `--limit` means
the number of instants. A legacy restore entry has no metadata field, so it is
logged and skipped instead of throwing.
- `copyNonArchivedInstants` stops at the limit and handles `restore` via
`readRestoreMetadata`.
- `TestExportCommand` drops the `--desc true --limit N` workaround and adds
tests for `--limit` in both orders, a completed restore, an LSM archive on the
current table version (crossing from active into archived instants in both
orders), and a legacy v6 archive.
The new tests all fail on master: the HoodieIOException on the history
directory and on `.hoodie/archived`, `Exported 3` for `--limit 2 --desc true`,
and `Unknown type of action restore`. They pass with this change.
`TestExportCommand`, `TestArchivedCommitsCommand` and `TestRestoresCommand`
pass locally with `-Pfunctional-tests -Dspark3.5 -Dscala-2.12` (13 tests), and
checkstyle passes.
### Impact
hudi-cli only. `export instants` no longer fails on the archive, and
`--limit` is now applied, so it can export fewer files than before when a limit
is given. The Spark `export_instants` procedure has the same archived-branch
problem. It is not touched here.
### Risk Level
low. The change is confined to the CLI export command and covered by
functional tests for both archive layouts.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
