ravjot28 opened a new pull request, #20590:
URL: https://github.com/apache/hudi/pull/20590
### Describe the issue this Pull Request addresses
Closes #20580
A table refresh removes the table's view from `FileSystemViewManager` and
closes it. A request resolved the view
separately for its freshness check, its sync, its handler and its final
check, so a refresh in between made it sync or
read a closed view (`NullPointerException`, HTTP 500) or answer from another
view than the one it checked.
`LAST_INSTANT`, `TIMELINE`, `LOAD_PARTITIONS` and `LOAD_ALL_PARTITIONS` read
the view with no coordination with a
refresh.
### Summary and Changelog
A request reads one view from its first check to its response, and a refresh
never closes a view a request is using.
- `FileSystemViewManager` hands out shared leases of a table's view
(`leaseFileSystemView`, `ViewLease`). A refresh,
`unregisterBasePath` or the manager's close removes the view and returns
at once; the view is closed when no lease of
it is held, otherwise by the release of its last lease. A view replaced
before a lease was taken is left to its
replacement.
- `RequestHandler`: a request that reads the view takes one lease for the
freshness check, the sync, the handler with
its response and the final check, and binds the leased view for its
handlers (`Handler#bindView`), which read it.
Handlers of one view still run in parallel; the view's monitor covers the
check and the sync only.
- The four endpoints above and the timeline read of `CREATE_MARKER` with
early conflict detection take the lease as
well.
- The leased view is bound per request thread instead of being passed
through every handler method, to keep the
handler signatures and the diff small.
- New test class `TestRequestHandlerViewReplacement`: a refresh at each
timeline read a request makes outside the
view's own sync (the positions come from a request that is not paused),
between the handler's view lookup and its
read, and before the handler traverses its result stream; handlers read
the leased view; two requests on one view
run their handlers at the same time; the four endpoints; a refresh returns
while a request holds the view; a lease
holder can refresh and close the manager without waiting; a retired view
with two leases closes with the last
release.
### Impact
No API change for clients and no configuration change.
`FileSystemViewManager` gains `leaseFileSystemView` and
`ViewLease`. A removed view may be closed slightly later, by the release of
the last request that uses it.
Open PR #20239 also changes `syncIfLocalViewBehind`; whichever lands second
needs a small rebase there.
`TestRequestHandlerViewReplacement#refreshDuringARequestClosesNoViewTheRequestReads`
finds the timeline reads of a
request at run time, so it does not depend on how many reads that method
makes.
**Backport:** please include this fix in the next 1.2.x patch release; the
race is present in 1.2.1. The commit does
not cherry-pick cleanly onto `release-1.2.1`: `RequestHandler` and `Handler`
conflict with changes made on master after
1.2.1 (#19946 and the timeline UI). I can open an adapted change against
`branch-1.2.x`.
### Risk Level
medium. The change touches the request path of every timeline server view
endpoint. Mitigation: the tests pause the
server-side view at each point a refresh can land, check that handlers of
one view still run concurrently, and the
existing timeline server and remote view suites pass.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources (scalastyle did not execute
in this Java-only reactor), with `mvn test -pl hudi-timeline-service -am
-Dtest=...` and the classes listed below:
- Without the main code change (the two tests that call the lease API left
out): 9 of the 10 remaining cases of
`TestRequestHandlerViewReplacement` fail, on "the view stays open while
the request holds it" (the refresh closed
a view in use; for `refreshDuringARequestClosesNoViewTheRequestReads` at
its first measured read) and on HTTP 500 in
`handlerReadsTheViewTheRequestHolds`.
- With the change: `TestRequestHandlerViewReplacement` 12/12,
`TestRequestHandler` 2/2, `TestTimelineService` 2/2,
`TestRemoteHoodieTableFileSystemView` 54/54,
`TestMarkerBasedEarlyConflictDetectionRunnable` 1/1,
`TestHoodieTableFileSystemView` 51/51,
`TestSpillableMapBasedFileSystemView` 51/51, `TestIncrementalFSViewSync` 5/5,
`TestSpillableMapBasedIncrementalFSViewSync` 5/5,
`TestPriorityBasedFileSystemView` 31/31 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]