weixiaonan1 opened a new pull request, #14833:
URL: https://github.com/apache/dolphinscheduler/pull/14833
<!--Thanks very much for contributing to Apache DolphinScheduler, we are
happy that you want to help us improve DolphinScheduler! -->
## Purpose of the pull request
This implementation allows to add listeners for monitoring and tracking the
workflow/task state using plugins without restarting the service.
1. Users can customize **listener plugins** by building a plugin JAR which
implemented the methods in ListenerPlugin and registering it in `Security
module`.
2. Users can create **listener instances** with the listener plugin and
choose the **listener events** to subscribe to in `Security module`.
3. The **listener events** monitored by the **listener instance** will be
pushed to the specified destination (message queues, external system
interfaces, etc.).
## Brief change log
Add new module: `dolphinscheduler-listener`.
## Verify this pull request
This change added tests and can be verified as follows:
Manually verified the change by testing locally and unit test will be added
later.
## Code Structure
`dolphinscheduler-listener`
- `dolphinscheduler-listener-common`: contains the interface definition of
**`ListenerPlugin`**, as well as definitions for various listener events. Users
only need to depend on this module to create their own listener plugins.
- `dolphinscheduler-listener-plugin`: examples of listener plugin
- dolphinscheduler-listener-logger: write listener events into log file
which is convenient to test the listener mechanism
- dolphinscheduler-listener-kafka: send listener events to Kafka
- `dolphinscheduler-listener-service`：core code of listener mechanism
- `ListenerPluginService`: manage the listener plugins along with the
listener instances
- `ListenerEventProducer`: interface definition for storing listener
events
- JdbcListenerEventProducer: save listener events into database
- `ListenerEventConsumer`: interface definition for
querying/updating/deleting listener events
- JdbcListenerEventConsumer: query/update/delete listener events
from database
- `ListenerEventPublishService`: save listener events asynchronously
- `ListenerInstancePostService`: query listener events and process them
## Listener Plugin & Listener Instance
1. create custom listener plugin
1. add dolphinscheduler-listener-common as dependency
2. implement methods in ListenerPlugin
3. package into a JAR file
2. install listener plugin
upload plugin JAR file and input the full class name of ListenerPlugin
in `Security module`.
3. update listener plugin
edit listener plugin and upload new plugin JAR file in `Security module`.
4. uninstall listener plugin
uninstall the listener plugin in `Security module`.(need to remove all
related listener instances first)
5. create listener instance (take LoggerPlugin as example)
input instance name, check event types to subscribe to, select listener
plugin and input plugin params in `Security module`.
6. update listener instance
update instance name, event types and plugin params in `Security module`.
7. delete listener instance
delete listener instance in `Security module`.
When install/update/uninstall listener plugins and create/update/delete
listener instances, the Api Server will send `RPC request` to `Alert Server` to
process.
ListenerPluginService injected into AlertServer:
## Listener Event:
Prerequisite: configure the listener type in application.yml (jdbc default)
1. ListenerEvent：there are 13 sorts of `ListenerEvent` defined in
`dolphinscheduler-listener-common`;
2. `Api Server` and `Master Server` will generate listener events and
publish to `ListenerEventPublishService` which save the events asynchronously
(To prevent the loss and disorder of listener events, we `persistently` store
the listener events).
3. ListenerEvents will be saved in database by default
`JdbcListenerEventProducer`. Users can implement their own producer to save
events in redis or other places;
4. Each listener instance has a corresponding `ListenerInstancePostService`
which takes listener events which handled by this instance and process them.
The events will be retrieved from database by default
`JdbcListenerEventConsumer`. Users can implement their own customer to process
events from redis or other places.
5. `ListenerInstancePostService` will perfom the following actions
periodically:
1. take events in chronological order first,
2. process them sequencially
1. If process successfully, invove `ListenerEventCosumer`.delete to
remove the event.
2. Otherwise, invove `ListenerEventCosumer`.upadte to update the
failure reason, and stop processing subsequent events.
## Database Design:
1. (modify) t_ds_plugin_define
id, plugin_name, plugin_type, plugin_params, plugin_location (new
column), plugin_class_name (new column), create_time, update_time
2. (added) t_ds_listener_plugin_instance
id, instance_name, plugin_define_id, plugin_instance_params,
listener_event_types, create_time, update_time
3. (added) t_ds_listener_event
id, content, post_status, event_type, log, plugin_instance_id,
create_time, update_time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
