Jzjsnow opened a new pull request, #4019:
URL: https://github.com/apache/amoro/pull/4019
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://amoro.apache.org/how-to-contribute/
2. If the PR is related to an issue in
https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title,
e.g., '[AMORO-XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][AMORO-XXXX] Your PR title ...'.
-->
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related
issue
-->
Close #4018
## Brief change log
<!--
Clearly describe the changes made in modules, classes, methods, etc.
-->
- Add a endpoint
/api/ams/v1/refresh/catalog/{catalog}/database/{database}/table/{table} for
triggering table refresh
- Add an ExternalEventService to receive and process posted external events
- Add support for specifying tables/catalogs that can trigger refreshes via
events in TableRuntimeRefreshExecutor:
- Two new variables added to TableRuntimeRefreshExecutor:
- `managedEventTriggerTables`: All tables configured using
event-triggered refreshes
- `pendingRefreshTables`: Tables awaiting refresh execution in the
next periodic schedule
In this PR, the event-triggered refresh process is as follows:
- If a table is configured for event-triggered refresh
(`self-optimizing.refresh.event-triggered=true`), its `tableIdentifier` is
added to `managedEventTriggerTables`
- If an event triggers refresh for a table in `managedEventTriggerTables`
(e.g., table commit event or manual trigger), its `tableIdentifier` is added to
`pendingRefreshTables`
- During the next scheduled execution of the `execute()` method for the
tableRuntime (at the fixed refresh interval `refresh-tables.interval`, default
1 minute), if the table remains in `pendingRefreshTables`, it undergoes
`loadTable()` and evaluation processes before being removed from the list;
otherwise, `loadTable()` and subsequent steps are skipped.
- If the table remains untriggered by events and reaches the fallback
interval (i.e., the time since the last refreshTime exceeds maxInterval), a
forced refresh and subsequent operations are executed.
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs /
not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
