This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new 0f43380e29 add two blog (#943)
0f43380e29 is described below
commit 0f43380e2924a3ce7f795296e9a7a1159a552dd1
Author: BaiJv <[email protected]>
AuthorDate: Fri Mar 1 10:01:46 2024 +0800
add two blog (#943)
* add two blog
* Optimize document
* fix: Remove external resources (#944)
related to https://whimsy.apache.org/site/project/dolphinscheduler
* add two blog file
* README.md modification I will create another PR
* Translate to English
* Revert this commit
---------
Co-authored-by: Jay Chung <[email protected]>
---
...ugin_development_for_enterprise_applications.md | 311 ++++++++++++++++++++
..._DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md | 282 +++++++++++++++++++
blog/img/2024-01-19/1.png | Bin 0 -> 456059 bytes
blog/img/2024-01-19/2.png | Bin 0 -> 438286 bytes
blog/img/2024-01-19/3.png | Bin 0 -> 412493 bytes
blog/img/2024-01-19/4.png | Bin 0 -> 1761368 bytes
blog/img/2024-01-19/5.png | Bin 0 -> 189147 bytes
blog/img/2024-01-19/6.png | Bin 0 -> 102593 bytes
...ugin_development_for_enterprise_applications.md | 312 +++++++++++++++++++++
..._DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md | 276 ++++++++++++++++++
config/blog/en-us/tech.json | 6 +
config/blog/en-us/tutorial.json | 6 +
config/blog/zh-cn/tech.json | 6 +
config/blog/zh-cn/tutorial.json | 6 +
14 files changed, 1205 insertions(+)
diff --git
a/blog/en-us/How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications.md
b/blog/en-us/How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications.md
new file mode 100644
index 0000000000..0e38834ae5
--- /dev/null
+++
b/blog/en-us/How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications.md
@@ -0,0 +1,311 @@
+In the 2.0.1 version of Apache DolphinScheduler, a plugin architecture
improvement was introduced to enhance the flexibility and user-friendliness of
the system. Components such as tasks, alert plugins, data sources, resource
storage, and registry centers are now designed as extension points. This allows
for targeted development of alert plugins to address specific alert
requirements in enterprise applications.
+**Current Version: 3.1.2**
+
+## Alert Plugin Development
+Let's take a look at the directory structure of the alert module:
+
+
+
+- dolphinscheduler-alert-api
+- This module serves as the core module for ALERT SPI. It defines the
interfaces for extending alert plugins and provides some basic code. The
AlertChannel and AlertChannelFactory interfaces are the key interfaces that
need to be implemented for developing alert plugins.
+- dolphinscheduler-alert-plugins
+- This module includes the official alert plugins provided by
DolphinScheduler. Currently, there are dozens of supported plugins, such as
Email, DingTalk, and Script.
+- dolphinscheduler-alert-server
+- This module represents the alert service, which is responsible for
registering alert plugins and sending alert messages via Netty.
+
+In this article, we will use the official HTTP alert plugin as an example to
demonstrate how to develop a plugin.
+- First, let's clarify the requirements. The HTTP alert plugin needs to send
requests via HTTP, and to do that, we need to determine the necessary
parameters. In the `HttpAlertConstants`, you can find the definitions of some
related parameters.
+
+```
+package org.apache.dolphinscheduler.plugin.alert.http;
+public final class HttpAlertConstants {
+ public static final String URL = "$t('url')";
+
+ public static final String NAME_URL = "url";
+
+ public static final String HEADER_PARAMS = "$t('headerParams')";
+
+ public static final String NAME_HEADER_PARAMS = "headerParams";
+
+...........................Omitting redundant code
+
+ private HttpAlertConstants() {
+ throw new UnsupportedOperationException("This is a utility class and
cannot be instantiated");
+ }
+}
+```
+
+- The corresponding parameters that need to be filled in for the alert
instance are shown in the image below.
+
+
+
+Here, parameters in the style of `$t('url')` can be added by editing the
+>dolphinscheduler-ui/src/locales/zh_CN/security.ts
+
+Once added, the frontend will automatically replace them. Similarly, the
English dictionary also needs to be updated to avoid errors when switching to
English.
+- In the `HttpAlertChannelFactory`, you need to implement the
`AlertChannelFactory`interface and its methods: `name`,`params`and`create`,The
first parameter of `InputParam.newBuilder` represents the displayed value, and
the second parameter represents the parameter name. You can use the constants
defined in `MailParamsConstants` that we mentioned earlier. After defining all
the parameters, add them to the `paramsList` and return.
+```
+@AutoService(AlertChannelFactory.class)
+public final class HttpAlertChannelFactory implements AlertChannelFactory {
+ @Override
+ public String name() {
+ return "Http";
+ }
+ @Override
+ public List<PluginParams> params() {
+ InputParam url = InputParam.newBuilder(HttpAlertConstants.NAME_URL,
HttpAlertConstants.URL)
+ .setPlaceholder("input request URL")
+ .addValidate(Validate.newBuilder()
+ .setRequired(true)
+ .build())
+ .build();
+ InputParam headerParams =
InputParam.newBuilder(HttpAlertConstants.NAME_HEADER_PARAMS,
HttpAlertConstants.HEADER_PARAMS)
+ .setPlaceholder("input request
headers as JSON format ")
+ .addValidate(Validate.newBuilder()
+
.setRequired(true)
+ .build())
+ .build();
+ InputParam bodyParams =
InputParam.newBuilder(HttpAlertConstants.NAME_BODY_PARAMS,
HttpAlertConstants.BODY_PARAMS)
+ .setPlaceholder("input request body
as JSON format ")
+ .addValidate(Validate.newBuilder()
+
.setRequired(false)
+ .build())
+ .build();
+...........................Omitting redundant code
+ return Arrays.asList(url, requestType, headerParams, bodyParams,
contentField);
+ }
+ @Override
+ public AlertChannel create() {
+ return new HttpAlertChannel();
+ }
+}
+```
+- In the `HttpAlertChannel`, you need to implement the `AlertChannel`
interface and its `process` method. The
`alertInfo.getAlertData().getAlertParams()` method can be used to retrieve the
parameters entered when creating the alert instance. Write the relevant code
here to send the request, and return an `AlertResult` object to indicate
whether the request was successfully sent or not.
+```
+public final class HttpAlertChannel implements AlertChannel {
+ @Override
+ public AlertResult process(AlertInfo alertInfo) {
+ AlertData alertData = alertInfo.getAlertData();
+ Map<String, String> paramsMap = alertInfo.getAlertParams();
+ if (null == paramsMap) {
+ return new AlertResult("false", "http params is null");
+ }
+ return new HttpSender(paramsMap).send(alertData.getContent());
+ }
+}
+```
+With that, the plugin development is complete. It's simple, isn't it? This is
how elegant and efficient decoupled code should be when designed with a
well-structured architecture.
+After completing the above development, start the alert service, and you will
be able to select the corresponding plugin when adding an alert instance.
+
+
+
+## Source Code Analysis
+When starting the alert service, you can see the information about registering
the alert plugins in the logs.
+
+
+
+Use this as a starting point to explore the relevant code for plugin
implementation.
+
+- In the `AlertPluginManager` of `dolphinscheduler-alert-server`, you can find
the content for registering the alert plugins. First, get all the classes that
implement `AlertChannelFactory.class`, and then iterate over them to obtain
instances of `AlertChannel`. Add these instances to the database and the
`channelKeyedById` map.
+```
+ private final Map<Integer, AlertChannel> channelKeyedById = new
HashMap<>();
+
+ @EventListener
+ public void installPlugin(ApplicationReadyEvent readyEvent) {
+ PrioritySPIFactory<AlertChannelFactory> prioritySPIFactory = new
PrioritySPIFactory<>(AlertChannelFactory.class);
+ for (Map.Entry<String, AlertChannelFactory> entry :
prioritySPIFactory.getSPIMap().entrySet()) {
+ String name = entry.getKey();
+ AlertChannelFactory factory = entry.getValue();
+ logger.info("Registering alert plugin: {} - {}", name,
factory.getClass());
+ final AlertChannel alertChannel = factory.create();
+ logger.info("Registered alert plugin: {} - {}", name,
factory.getClass());
+ final List<PluginParams> params = new
ArrayList<>(factory.params());
+ params.add(0, warningTypeParams);
+ final String paramsJson =
PluginParamsTransfer.transferParamsToJson(params);
+ final PluginDefine pluginDefine = new PluginDefine(name,
PluginType.ALERT.getDesc(), paramsJson);
+ final int id = pluginDao.addOrUpdatePluginDefine(pluginDefine);
+ channelKeyedById.put(id, alertChannel);
+ }
+ }
+```
+- After developing and registering the plugins, a polling thread is needed to
iterate and perform actions for querying and sending messages. The `run` method
of `AlertSenderService` handles this.
+
+```
+@Override
+public void run() {
+ logger.info("alert sender started");
+ while (!ServerLifeCycleManager.isStopped()) {
+ try {
+ List<Alert> alerts = alertDao.listPendingAlerts();
+ AlertServerMetrics.registerPendingAlertGauge(alerts::size);
+ this.send(alerts);
+ ThreadUtils.sleep(Constants.SLEEP_TIME_MILLIS * 5L);
+ } catch (Exception e) {
+ logger.error("alert sender thread error", e);
+ }
+ }
+}
+```
+- The key method is `this.send(alerts)`. Iterate over the `Alert` instances,
retrieve the instances of alert plugins, and pass them along with the alert
parameters to `this.alertResultHandler(instance, alertData)`. Finally, update
the status of the alert message.
+
+```
+public void send(List<Alert> alerts) {
+ for (Alert alert : alerts) {
+ // get alert group from alert
+ int alertId = Optional.ofNullable(alert.getId()).orElse(0);
+ int alertGroupId =
Optional.ofNullable(alert.getAlertGroupId()).orElse(0);
+ List<AlertPluginInstance> alertInstanceList =
alertDao.listInstanceByAlertGroupId(alertGroupId);
+ if (CollectionUtils.isEmpty(alertInstanceList)) {
+ logger.error("send alert msg fail,no bind plugin instance.");
+ List<AlertResult> alertResults = Lists.newArrayList(new
AlertResult("false",
+ "no bind plugin instance"));
+ alertDao.updateAlert(AlertStatus.EXECUTION_FAILURE,
JSONUtils.toJsonString(alertResults), alertId);
+ continue;
+ }
+ AlertData alertData = AlertData.builder()
+ .id(alertId)
+ .content(alert.getContent())
+ .log(alert.getLog())
+ .title(alert.getTitle())
+ .warnType(alert.getWarningType().getCode())
+ .alertType(alert.getAlertType().getCode())
+ .build();
+
+ int sendSuccessCount = 0;
+ List<AlertResult> alertResults = new ArrayList<>();
+ for (AlertPluginInstance instance : alertInstanceList) {
+ AlertResult alertResult = this.alertResultHandler(instance,
alertData);
+ if (alertResult != null) {
+ AlertStatus sendStatus =
Boolean.parseBoolean(String.valueOf(alertResult.getStatus()))
+ ? AlertStatus.EXECUTION_SUCCESS
+ : AlertStatus.EXECUTION_FAILURE;
+ alertDao.addAlertSendStatus(sendStatus,
JSONUtils.toJsonString(alertResult), alertId,
+ instance.getId());
+ if (sendStatus.equals(AlertStatus.EXECUTION_SUCCESS)) {
+ sendSuccessCount++;
+ AlertServerMetrics.incAlertSuccessCount();
+ } else {
+ AlertServerMetrics.incAlertFailCount();
+ }
+ alertResults.add(alertResult);
+ }
+ }
+ AlertStatus alertStatus = AlertStatus.EXECUTION_SUCCESS;
+ if (sendSuccessCount == 0) {
+ alertStatus = AlertStatus.EXECUTION_FAILURE;
+ } else if (sendSuccessCount < alertInstanceList.size()) {
+ alertStatus = AlertStatus.EXECUTION_PARTIAL_SUCCESS;
+ }
+ alertDao.updateAlert(alertStatus,
JSONUtils.toJsonString(alertResults), alertId);
+ }
+}
+```
+- In the `alertResultHandler` method, use
`alertPluginManager.getAlertChannel(instance.getPluginDefineId())` to retrieve
an instance of `AlertChannel`. Remember when we put the `AlertChannel`
instances into the `channelKeyedById` map during the registration of alert
plugins?
+```
+public Optional<AlertChannel> getAlertChannel(int id) {
+ return Optional.ofNullable(channelKeyedById.get(id));
+}
+```
+- Then, build an `AlertInfo` object and use `CompletableFuture.supplyAsync()`
to asynchronously execute `alertChannel.process(alertInfo)`. Obtain the
returned `AlertResult` using `future.get()` and return it.
+```
+private @Nullable AlertResult alertResultHandler(AlertPluginInstance instance,
AlertData alertData) {
+ String pluginInstanceName = instance.getInstanceName();
+ int pluginDefineId = instance.getPluginDefineId();
+ Optional<AlertChannel> alertChannelOptional =
alertPluginManager.getAlertChannel(instance.getPluginDefineId());
+ if (!alertChannelOptional.isPresent()) {
+ String message = String.format("Alert Plugin %s send error: the
channel doesn't exist, pluginDefineId: %s",
+ pluginInstanceName,
+ pluginDefineId);
+ logger.error("Alert Plugin {} send error : not found plugin {}",
pluginInstanceName, pluginDefineId);
+ return new AlertResult("false", message);
+ }
+ AlertChannel alertChannel = alertChannelOptional.get();
+
+ Map<String, String> paramsMap =
JSONUtils.toMap(instance.getPluginInstanceParams());
+ String instanceWarnType = WarningType.ALL.getDescp();
+
+ if (paramsMap != null) {
+ instanceWarnType =
paramsMap.getOrDefault(AlertConstants.NAME_WARNING_TYPE,
WarningType.ALL.getDescp());
+ }
+
+ WarningType warningType = WarningType.of(instanceWarnType);
+
+ if (warningType == null) {
+ String message = String.format("Alert Plugin %s send error : plugin
warnType is null", pluginInstanceName);
+ logger.error("Alert Plugin {} send error : plugin warnType is null",
pluginInstanceName);
+ return new AlertResult("false", message);
+ }
+
+ boolean sendWarning = false;
+ switch (warningType) {
+ case ALL:
+ sendWarning = true;
+ break;
+ case SUCCESS:
+ if (alertData.getWarnType() == WarningType.SUCCESS.getCode()) {
+ sendWarning = true;
+ }
+ break;
+ case FAILURE:
+ if (alertData.getWarnType() == WarningType.FAILURE.getCode()) {
+ sendWarning = true;
+ }
+ break;
+ default:
+ }
+
+ if (!sendWarning) {
+ logger.info(
+ "Alert Plugin {} send ignore warning type not match: plugin
warning type is {}, alert data warning type is {}",
+ pluginInstanceName, warningType.getCode(),
alertData.getWarnType());
+ return null;
+ }
+
+ AlertInfo alertInfo = AlertInfo.builder()
+ .alertData(alertData)
+ .alertParams(paramsMap)
+ .alertPluginInstanceId(instance.getId())
+ .build();
+ int waitTimeout = alertConfig.getWaitTimeout();
+ try {
+ AlertResult alertResult;
+ if (waitTimeout <= 0) {
+ if (alertData.getAlertType() == AlertType.CLOSE_ALERT.getCode()) {
+ alertResult = alertChannel.closeAlert(alertInfo);
+ } else {
+ alertResult = alertChannel.process(alertInfo);
+ }
+ } else {
+ CompletableFuture<AlertResult> future;
+ if (alertData.getAlertType() == AlertType.CLOSE_ALERT.getCode()) {
+ future = CompletableFuture.supplyAsync(() ->
alertChannel.closeAlert(alertInfo));
+ } else {
+ future = CompletableFuture.supplyAsync(() ->
alertChannel.process(alertInfo));
+ }
+ alertResult = future.get(waitTimeout, TimeUnit.MILLISECONDS);
+ }
+ if (alertResult == null) {
+ throw new RuntimeException("Alert result cannot be null");
+ }
+ return alertResult;
+ } catch (InterruptedException e) {
+ logger.error("send alert error alert data id :{},", alertData.getId(),
e);
+ Thread.currentThread().interrupt();
+ return new AlertResult("false", e.getMessage());
+ } catch (Exception e) {
+ logger.error("send alert error alert data id :{},", alertData.getId(),
e);
+ return new AlertResult("false", e.getMessage());
+ }
+}
+```
+
+In summary, the sequence diagram below illustrates the registration of plugins
and the sending of messages.
+
+
+
+This covers the main implementation code for alert plugins. As you can see,
the source code doesn't appear to be too complex or difficult to understand.
So, take some time to explore the source code. In the future, you can also
contribute to open-source projects and create such excellent software.
+
+References
+>[[Feature] Alert Plugin Design · Issue #3049 · apache/dolphinscheduler
(github.com)](https://github.com/apache/dolphinscheduler/issues/3049)
+
+>[alert
(apache.org)](https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/contribute/backend/spi/alert.html)
\ No newline at end of file
diff --git
a/blog/en-us/Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md
b/blog/en-us/Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md
new file mode 100644
index 0000000000..15862e9e8e
--- /dev/null
+++
b/blog/en-us/Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md
@@ -0,0 +1,282 @@
+[Apache
DolphinScheduler](https://dolphinscheduler.apache.org/zh-cn/docs/3.1.2/guide/upgrade/upgrade)
+**By referring to the official upgrade documentation, it is known that upgrade
scripts are provided. If it is a minor version upgrade, executing the script
should suffice. However, when upgrading across multiple major versions, various
issues can still arise. Therefore, a summary of these issues is provided.**
+
+old version：1.3.4
+new version：3.1.2
+
+## 1. Error "IllegalArgumentException: Failed to specify server's Kerberos
principal name" when using Resource Center after the upgrade
+The Resource Center is configured to use HDFS with Kerberos authentication
enabled.
+
+#### Solution:
+Edit `dolphinscheduler/api-server/conf/hdfs-site.xml` and add the following
content:
+```xml
+<property>
+ <name>dfs.namenode.kerberos.principal.pattern</name>
+ <value>*</value>
+</property>
+```
+
+## 2. Error "Log not found" when viewing task instance logs after the upgrade
+**Upon checking the error message and comparing the directory structure and
log paths in the new version, it is found that the log path has been changed in
the new version.**
+**The old log path was located under `/logs/`, while the new log path is
`/worker-server/logs/`.**
+**Therefore, the directory needs to be modified accordingly.**
+
+#### Solution:
+Execute the following SQL statement to modify the log path:
+```sql
+update t_ds_task_instance set
log_path=replace(log_path,'/logs/','/worker-server/logs/');
+```
+Then, copy the original log files to the new log path:
+```sql
+cp -r {old_dolphinscheduler_directory}/logs/[1-9]*
{new_dolphinscheduler_directory}/worker-server/logs/*
+```
+
+## 3. Error when creating workflows after the upgrade
+Upon checking the error message, it is found that the initial values of the
primary keys for `t_ds_process_definition_log` and `t_ds_process_definition`
are inconsistent.
+To resolve this, the primary keys need to be made consistent.
+
+#### Solution:
+Execute the following SQL statement:
+```sql
+# Retrieve the auto-increment value of the primary key
+select AUTO_INCREMENT FROM information_schema.TABLES WHERE TABLE_SCHEMA =
'dolphinscheduler' AND TABLE_NAME = 't_ds_process_definition' limit 1
+
+# Replace {max_id} with the above result and execute the statement
+alter table dolphinscheduler_bak1.t_ds_process_definition_log auto_increment =
{max_id};
+```
+
+## 4. Task instance list is empty after the upgrade
+Check the SQL query in
`dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapper.xml`
file under the select id="queryTaskInstanceListPaging" section.
+```sql
+ select
+ <include refid="baseSqlV2">
+ <property name="alias" value="instance"/>
+ </include>
+ ,
+ process.name as process_instance_name
+ from t_ds_task_instance instance
+ left join t_ds_task_definition_log define on
define.code=instance.task_code and
define.version=instance.task_definition_version
+ left join t_ds_process_instance process on
process.id=instance.process_instance_id
+ where define.project_code = #{projectCode}
+ <if test="startTime != null">
+ and instance.start_time <![CDATA[ >=]]> #{startTime}
+ </if>
+ ......Omitting redundant code
+```
+**The SQL query for querying the task instance list is associated with the
`t_ds_task_definition_log` table. After inspection, it was found that the join
condition `define.code = instance.task_code` cannot be matched.**
+**Considering the query condition `define.project_code = #{projectCode}`, it
can be inferred that the purpose of joining the `t_ds_task_definition_log`
table is mainly to filter by `projectCode`. Let's modify the SQL accordingly.**
+#### Solution:
+```sql
+ select
+ <include refid="baseSqlV2">
+ <property name="alias" value="instance"/>
+ </include>
+ ,
+ process.name as process_instance_name
+ from t_ds_task_instance instance
+-- left join t_ds_task_definition_log define
+-- on define.code=instance.task_code and
+--
define.version=instance.task_definition_version
+ join t_ds_process_instance process
+ on process.id=instance.process_instance_id
+ join t_ds_process_definition define
+ on define.code=process.process_definition_code
+ where define.project_code = #{projectCode}
+ <if test="startTime != null">
+ and instance.start_time <![CDATA[ >=]]> #{startTime}
+ </if>
+ ......Omitting redundant code
+```
+**Modify the SQL query to directly use the `t_ds_process_definition` table for
the association, as it also has the `project_code` field for filtering.**
+
+## 5. NullPointerException during the execution of the upgrade script
+### 5.1 Analysis of the logs led to line 517 in UpgradeDao.java
+
+```java
+513 if (TASK_TYPE_SUB_PROCESS.equals(taskType)) {
+514 JsonNode jsonNodeDefinitionId =
param.get("processDefinitionId");
+515 if (jsonNodeDefinitionId != null) {
+516 param.put("processDefinitionCode",
+517
processDefinitionMap.get(jsonNodeDefinitionId.asInt()).getCode());
+518 param.remove("processDefinitionId");
+519 }
+520 }
+```
+
+**Upon examining the code, it is evident that
`processDefinitionMap.get(jsonNodeDefinitionId.asInt())` returns null. Add a
null check to skip the null value and print the relevant information for
verification after the upgrade.**
+
+#### Solution:
+After modification:
+```java
+ if (jsonNodeDefinitionId != null) {
+ if (processDefinitionMap.get(jsonNodeDefinitionId.asInt()) !=
null) {
+
param.put("processDefinitionCode",processDefinitionMap.get(jsonNodeDefinitionId.asInt()).getCode());
+ param.remove("processDefinitionId");
+ } else {
+ logger.error("*******************error");
+ logger.error("*******************param:" + param);
+ logger.error("*******************jsonNodeDefinitionId:" +
jsonNodeDefinitionId);
+ }
+ }
+```
+
+### 5.2 Analysis of the logs led to line 675 in UpgradeDao.java
+```java
+669 if (mapEntry.isPresent()) {
+670 Map.Entry<Long, Map<String, Long>>
processCodeTaskNameCodeEntry = mapEntry.get();
+671 dependItem.put("definitionCode",
processCodeTaskNameCodeEntry.getKey());
+672 String depTasks =
dependItem.get("depTasks").asText();
+673 long taskCode =
+674 "ALL".equals(depTasks) ||
processCodeTaskNameCodeEntry.getValue() == null ? 0L
+675 :
processCodeTaskNameCodeEntry.getValue().get(depTasks);
+676 dependItem.put("depTaskCode", taskCode);
+677 }
+```
+Upon examining the code, it is evident that
`processCodeTaskNameCodeEntry.getValue().get(depTasks)` returns null. Modify
the logic to assign a value and print the relevant log only if it is not null.
+
+#### Solution:
+After modification:
+```java
+ long taskCode =0;
+ if (processCodeTaskNameCodeEntry.getValue() != null
+ &&processCodeTaskNameCodeEntry.getValue().get(depTasks)!=null){
+ taskCode =processCodeTaskNameCodeEntry.getValue().get(depTasks);
+ }else{
+ logger.error("******************** depTasks:"+depTasks);
+ logger.error("******************** taskCode not in
"+JSONUtils.toJsonString(processCodeTaskNameCodeEntry));
+ }
+ dependItem.put("depTaskCode", taskCode);
+```
+## 6. Login failure after integrating LDAP, unknown field name for email
+LDAP integration can be configured in `api-server/conf/application.yaml`.
+```yaml
+security:
+ authentication:
+ # Authentication types (supported types: PASSWORD,LDAP)
+ type: LDAP
+ # IF you set type `LDAP`, below config will be effective
+ ldap:
+ # ldap server config
+ urls: xxx
+ base-dn: xxx
+ username: xxx
+ password: xxx
+ user:
+ # admin userId when you use LDAP login
+ admin: xxx
+ identity-attribute: xxx
+ email-attribute: xxx
+ # action when ldap user is not exist (supported types: CREATE,DENY)
+ not-exist-action: CREATE
+```
+**To successfully integrate LDAP, the following fields need to be correctly
filled in the configuration: `urls`, `base-dn`, `username`, `password`,
`identity`, and `email`. If the email field name is unknown, follow the steps
below, leaving the email field empty for now:**
+**Start the service and attempt to log in with an LDAP user.**
+#### Solution:
+** The LDAP authentication code is located in
`dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/security/impl/ldap/LdapService.java`
under the `ldapLogin()` method.**
+
+```java
+ ctx = new InitialLdapContext(searchEnv, null);
+ SearchControls sc = new SearchControls();
+ sc.setReturningAttributes(new String[]{ldapEmailAttribute});
+ sc.setSearchScope(SearchControls.SUBTREE_SCOPE);
+ EqualsFilter filter = new EqualsFilter(ldapUserIdentifyingAttribute,
userId);
+ NamingEnumeration<SearchResult> results = ctx.search(ldapBaseDn,
filter.toString(), sc);
+ if (results.hasMore()) {
+ // get the users DN (distinguishedName) from the result
+ SearchResult result = results.next();
+ NamingEnumeration<? extends Attribute> attrs =
result.getAttributes().getAll();
+ while (attrs.hasMore()) {
+ // Open another connection to the LDAP server with the found
DN and the password
+ searchEnv.put(Context.SECURITY_PRINCIPAL,
result.getNameInNamespace());
+ searchEnv.put(Context.SECURITY_CREDENTIALS, userPwd);
+ try {
+ new InitialDirContext(searchEnv);
+ } catch (Exception e) {
+ logger.warn("invalid ldap credentials or ldap search
error", e);
+ return null;
+ }
+ Attribute attr = attrs.next();
+ if (attr.getID().equals(ldapEmailAttribute)) {
+ return (String) attr.get();
+ }
+ }
+ }
+```
+
+Comment out the 3 line that filters based on the field filled.
+```java
+// sc.setReturningAttributes(new String[]{ldapEmailAttribute});
+```
+After executing, the 10 line will return all fields.
+```java
+NamingEnumeration<? extends Attribute> attrs = result.getAttributes().getAll();
+```
+Find the email field through printing or debugging and fill it in the
configuration file.
+Uncomment the previously commented line of code.
+Restart the service to enable successful LDAP integration and login.
+
+## 7. Authorization of resource files by administrators for ordinary users
does not take effect
+After multiple tests, it was found that ordinary users can only see resource
files that belong to them, even after being granted authorization by
administrators.
+
+#### Solution:
+**Modify the `listAuthorizedResource()` method in the file
`dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/permission/ResourcePermissionCheckServiceImpl.java`
to return `relationResources` instead of the current collection.**
+```java
+ @Override
+ public Set<Integer> listAuthorizedResource(int userId, Logger logger) {
+ List<Resource> relationResources;
+ if (userId == 0) {
+ relationResources = new ArrayList<>();
+ } else {
+ // query resource relation
+ List<Integer> resIds =
resourceUserMapper.queryResourcesIdListByUserIdAndPerm(userId, 0);
+ relationResources = CollectionUtils.isEmpty(resIds) ? new
ArrayList<>() : resourceMapper.queryResourceListById(resIds);
+ }
+ List<Resource> ownResourceList =
resourceMapper.queryResourceListAuthored(userId, -1);
+ relationResources.addAll(ownResourceList);
+ return
relationResources.stream().map(Resource::getId).collect(toSet()); // Resolve
the issue of invalid resource file authorization.
+// return
ownResourceList.stream().map(Resource::getId).collect(toSet());
+ }
+```
+Check the change log of the new version and find that this bug has been fixed
in version 3.1.3.
+https://github.com/apache/dolphinscheduler/pull/13318
+## 8.Kerberos expiration issue.
+Due to the expiration time set in the Kerberos configuration, the HDFS
resources in the Resource Center will become inaccessible after a certain
period of time. The best solution is to add relevant logic for scheduled
credential update.
+#### Solution:
+Add a method in the file
`dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/utils/CommonUtils.java`.```java
+ ```java
+ /**
+ * * Scheduled credential update.
+ */
+ private static void startCheckKeytabTgtAndReloginJob() {
+ // Daily loop, scheduled credential update.
+ Executors.newScheduledThreadPool(1).scheduleWithFixedDelay(() -> {
+ try {
+
UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();
+ logger.warn("Check Kerberos Tgt And Relogin From Keytab
Finish.");
+ } catch (IOException e) {
+ logger.error("Check Kerberos Tgt And Relogin From Keytab
Error", e);
+ }
+ }, 0, 1, TimeUnit.DAYS);
+ logger.info("Start Check Keytab TGT And Relogin Job Success.");
+ }
+```
+Then, call it before the `loadKerberosConf` method in that file returns true.
+```java
+public static boolean loadKerberosConf(String javaSecurityKrb5Conf, String
loginUserKeytabUsername,
+ String loginUserKeytabPath,
Configuration configuration) throws IOException {
+ if (CommonUtils.getKerberosStartupState()) {
+ System.setProperty(Constants.JAVA_SECURITY_KRB5_CONF,
StringUtils.defaultIfBlank(javaSecurityKrb5Conf,
+
PropertyUtils.getString(Constants.JAVA_SECURITY_KRB5_CONF_PATH)));
+ configuration.set(Constants.HADOOP_SECURITY_AUTHENTICATION,
Constants.KERBEROS);
+ UserGroupInformation.setConfiguration(configuration);
+ UserGroupInformation.loginUserFromKeytab(
+ StringUtils.defaultIfBlank(loginUserKeytabUsername,
+
PropertyUtils.getString(Constants.LOGIN_USER_KEY_TAB_USERNAME)),
+ StringUtils.defaultIfBlank(loginUserKeytabPath,
+
PropertyUtils.getString(Constants.LOGIN_USER_KEY_TAB_PATH)));
+ startCheckKeytabTgtAndReloginJob(); // call here
+ return true;
+ }
+ return false;
+ }
+```
\ No newline at end of file
diff --git a/blog/img/2024-01-19/1.png b/blog/img/2024-01-19/1.png
new file mode 100644
index 0000000000..60836fa514
Binary files /dev/null and b/blog/img/2024-01-19/1.png differ
diff --git a/blog/img/2024-01-19/2.png b/blog/img/2024-01-19/2.png
new file mode 100644
index 0000000000..e7608e640f
Binary files /dev/null and b/blog/img/2024-01-19/2.png differ
diff --git a/blog/img/2024-01-19/3.png b/blog/img/2024-01-19/3.png
new file mode 100644
index 0000000000..e52b515d31
Binary files /dev/null and b/blog/img/2024-01-19/3.png differ
diff --git a/blog/img/2024-01-19/4.png b/blog/img/2024-01-19/4.png
new file mode 100644
index 0000000000..17a9f2b238
Binary files /dev/null and b/blog/img/2024-01-19/4.png differ
diff --git a/blog/img/2024-01-19/5.png b/blog/img/2024-01-19/5.png
new file mode 100644
index 0000000000..9bc0032da4
Binary files /dev/null and b/blog/img/2024-01-19/5.png differ
diff --git a/blog/img/2024-01-19/6.png b/blog/img/2024-01-19/6.png
new file mode 100644
index 0000000000..624de2477a
Binary files /dev/null and b/blog/img/2024-01-19/6.png differ
diff --git
a/blog/zh-cn/How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications.md
b/blog/zh-cn/How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications.md
new file mode 100644
index 0000000000..d4d71ffa99
--- /dev/null
+++
b/blog/zh-cn/How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications.md
@@ -0,0 +1,312 @@
+在Apache DolphinScheduler的2.0.1版本
加入了插件化架构改进，将任务、告警组件、数据源、资源存储、注册中心等都将被设计为扩展点，以此来提高 Apache DolphinScheduler
本身的灵活性和友好性。在企业级应用中根据不同公司的告警需求可能各有不同，针对性的告警插件开发可以很好的解决这一痛点。
+
+**当前版本：3.1.2**
+
+## 告警插件开发
+先来看下alert目录的结构
+
+
+
+- dolphinscheduler-alert-api
+- 该模块是 ALERT SPI 的核心模块，该模块定义了告警插件扩展的接口以及一些基础代码，其中 AlertChannel 和
AlertChannelFactory 是告警插件开发需要实现的接口类
+- dolphinscheduler-alert-plugins
+- 该模块包含了官方提供的告警插件，目前我们已经支持数十种插件，如 Email、DingTalk、Script等
+- dolphinscheduler-alert-server
+- 告警服务模块,主要功能包括注册告警插件,Netty告警消息发送等
+
+
+本文以官方的http告警插件为例讲解如何进行插件开发
+- 首先明确需求,http告警插件需要通过http发送请求,发送请求首先需要确定哪些参数.在 `HttpAlertConstants`
可以看到有定义一些相关参数
+```
+package org.apache.dolphinscheduler.plugin.alert.http;
+public final class HttpAlertConstants {
+ public static final String URL = "$t('url')";
+
+ public static final String NAME_URL = "url";
+
+ public static final String HEADER_PARAMS = "$t('headerParams')";
+
+ public static final String NAME_HEADER_PARAMS = "headerParams";
+
+...........................省略多余代码
+
+ private HttpAlertConstants() {
+ throw new UnsupportedOperationException("This is a utility class and
cannot be instantiated");
+ }
+}
+```
+
+- 对应此处告警实例需要填写的参数
+
+
+
+其中 $t('url') 样式的参数可以通过编辑
+>dolphinscheduler-ui/src/locales/zh_CN/security.ts
+
+添加对应的参数,前端收到后会自动替换,同样的英文字典也需要替换,不然切换英文时会报错
+
+-
在`HttpAlertChannelFactory`需要实现`AlertChannelFactory`并实现它的方法`name`,`params`和`create`。其中`InputParam.newBuilder`的第一个参数是显示的值,第二个参数是参数名，这里用我们前面在`MailParamsConstants`写好的常量。所有参数写好后添加到`paramsList`后返回
+```
+@AutoService(AlertChannelFactory.class)
+public final class HttpAlertChannelFactory implements AlertChannelFactory {
+ @Override
+ public String name() {
+ return "Http";
+ }
+ @Override
+ public List<PluginParams> params() {
+ InputParam url = InputParam.newBuilder(HttpAlertConstants.NAME_URL,
HttpAlertConstants.URL)
+ .setPlaceholder("input request URL")
+ .addValidate(Validate.newBuilder()
+ .setRequired(true)
+ .build())
+ .build();
+ InputParam headerParams =
InputParam.newBuilder(HttpAlertConstants.NAME_HEADER_PARAMS,
HttpAlertConstants.HEADER_PARAMS)
+ .setPlaceholder("input request
headers as JSON format ")
+ .addValidate(Validate.newBuilder()
+
.setRequired(true)
+ .build())
+ .build();
+ InputParam bodyParams =
InputParam.newBuilder(HttpAlertConstants.NAME_BODY_PARAMS,
HttpAlertConstants.BODY_PARAMS)
+ .setPlaceholder("input request body
as JSON format ")
+ .addValidate(Validate.newBuilder()
+
.setRequired(false)
+ .build())
+ .build();
+...........................省略多余代码
+ return Arrays.asList(url, requestType, headerParams, bodyParams,
contentField);
+ }
+ @Override
+ public AlertChannel create() {
+ return new HttpAlertChannel();
+ }
+}
+```
+-
在`HttpAlertChannel`需要实现`AlertChannel`并实现`process`方法,其中`alertInfo.getAlertData().getAlertParams()`可以拿到在创建告警实例时填写的参数,在此处编写相关代码发送请求后,需要返回`AlertResult`对象用来标记请求发送or失败
+```
+public final class HttpAlertChannel implements AlertChannel {
+ @Override
+ public AlertResult process(AlertInfo alertInfo) {
+ AlertData alertData = alertInfo.getAlertData();
+ Map<String, String> paramsMap = alertInfo.getAlertParams();
+ if (null == paramsMap) {
+ return new AlertResult("false", "http params is null");
+ }
+ return new HttpSender(paramsMap).send(alertData.getContent());
+ }
+}
+```
+至此插件开发就完成的,是不是很简单：）设计优秀架构合理的代码就应该是这样优雅高效解耦合.
+完成以上开发后,启动告警服务,就可以在添加告警实例时选择对应的插件了
+
+
+
+
+## 源码解读
+在启动告警服务时,可以在日志看到有注册告警插件的信息
+
+
+
+以此为切入口来探索插件实现的相关代码
+
+- 在dolphinscheduler-alert-server的`AlertPluginManager`的 installPlugin
方法可以看到注册告警插件的内容,这里先获取所有实现了`AlertChannelFactory.class`的类,遍历后获取`AlertChannel`的实例,添加到数据库和`channelKeyedById`Map
+```
+ private final Map<Integer, AlertChannel> channelKeyedById = new
HashMap<>();
+
+ @EventListener
+ public void installPlugin(ApplicationReadyEvent readyEvent) {
+ PrioritySPIFactory<AlertChannelFactory> prioritySPIFactory = new
PrioritySPIFactory<>(AlertChannelFactory.class);
+ for (Map.Entry<String, AlertChannelFactory> entry :
prioritySPIFactory.getSPIMap().entrySet()) {
+ String name = entry.getKey();
+ AlertChannelFactory factory = entry.getValue();
+ logger.info("Registering alert plugin: {} - {}", name,
factory.getClass());
+ final AlertChannel alertChannel = factory.create();
+ logger.info("Registered alert plugin: {} - {}", name,
factory.getClass());
+ final List<PluginParams> params = new
ArrayList<>(factory.params());
+ params.add(0, warningTypeParams);
+ final String paramsJson =
PluginParamsTransfer.transferParamsToJson(params);
+ final PluginDefine pluginDefine = new PluginDefine(name,
PluginType.ALERT.getDesc(), paramsJson);
+ final int id = pluginDao.addOrUpdatePluginDefine(pluginDefine);
+ channelKeyedById.put(id, alertChannel);
+ }
+ }
+```
+- 完成插件的开发和注册后,需要有个轮询线程来遍历查询需要发送的消息和完成发送的动作,在`AlertSenderService`的`run`方法完成了这些
+```
+@Override
+public void run() {
+ logger.info("alert sender started");
+ while (!ServerLifeCycleManager.isStopped()) {
+ try {
+ List<Alert> alerts = alertDao.listPendingAlerts();
+ AlertServerMetrics.registerPendingAlertGauge(alerts::size);
+ this.send(alerts);
+ ThreadUtils.sleep(Constants.SLEEP_TIME_MILLIS * 5L);
+ } catch (Exception e) {
+ logger.error("alert sender thread error", e);
+ }
+ }
+}
+```
+- 关键方法是`this.send(alerts)`,这里遍历`Alert`后获取告警插件的实例集合,在
`this.alertResultHandler(instance, alertData)`传入插件实例对象和告警参数,最后更新这条告警消息的状态
+```
+public void send(List<Alert> alerts) {
+ for (Alert alert : alerts) {
+ // get alert group from alert
+ int alertId = Optional.ofNullable(alert.getId()).orElse(0);
+ int alertGroupId =
Optional.ofNullable(alert.getAlertGroupId()).orElse(0);
+ List<AlertPluginInstance> alertInstanceList =
alertDao.listInstanceByAlertGroupId(alertGroupId);
+ if (CollectionUtils.isEmpty(alertInstanceList)) {
+ logger.error("send alert msg fail,no bind plugin instance.");
+ List<AlertResult> alertResults = Lists.newArrayList(new
AlertResult("false",
+ "no bind plugin instance"));
+ alertDao.updateAlert(AlertStatus.EXECUTION_FAILURE,
JSONUtils.toJsonString(alertResults), alertId);
+ continue;
+ }
+ AlertData alertData = AlertData.builder()
+ .id(alertId)
+ .content(alert.getContent())
+ .log(alert.getLog())
+ .title(alert.getTitle())
+ .warnType(alert.getWarningType().getCode())
+ .alertType(alert.getAlertType().getCode())
+ .build();
+
+ int sendSuccessCount = 0;
+ List<AlertResult> alertResults = new ArrayList<>();
+ for (AlertPluginInstance instance : alertInstanceList) {
+ AlertResult alertResult = this.alertResultHandler(instance,
alertData);
+ if (alertResult != null) {
+ AlertStatus sendStatus =
Boolean.parseBoolean(String.valueOf(alertResult.getStatus()))
+ ? AlertStatus.EXECUTION_SUCCESS
+ : AlertStatus.EXECUTION_FAILURE;
+ alertDao.addAlertSendStatus(sendStatus,
JSONUtils.toJsonString(alertResult), alertId,
+ instance.getId());
+ if (sendStatus.equals(AlertStatus.EXECUTION_SUCCESS)) {
+ sendSuccessCount++;
+ AlertServerMetrics.incAlertSuccessCount();
+ } else {
+ AlertServerMetrics.incAlertFailCount();
+ }
+ alertResults.add(alertResult);
+ }
+ }
+ AlertStatus alertStatus = AlertStatus.EXECUTION_SUCCESS;
+ if (sendSuccessCount == 0) {
+ alertStatus = AlertStatus.EXECUTION_FAILURE;
+ } else if (sendSuccessCount < alertInstanceList.size()) {
+ alertStatus = AlertStatus.EXECUTION_PARTIAL_SUCCESS;
+ }
+ alertDao.updateAlert(alertStatus,
JSONUtils.toJsonString(alertResults), alertId);
+ }
+}
+```
+-
在`alertResultHandler`用`alertPluginManager.getAlertChannel(instance.getPluginDefineId())`获取`AlertChannel`实例.还记得前面注册告警插件时往`channelKeyedById`里put的`AlertChannel`实例的动作吗?
+```
+public Optional<AlertChannel> getAlertChannel(int id) {
+ return Optional.ofNullable(channelKeyedById.get(id));
+}
+```
+-
然后构建`AlertInfo`对象,通过`CompletableFuture.supplyAsync()`来异步回调执行`alertChannel.process(alertInfo)`,用`future.get()`获得回调执行返回的`AlertResult`再return
+```
+private @Nullable AlertResult alertResultHandler(AlertPluginInstance instance,
AlertData alertData) {
+ String pluginInstanceName = instance.getInstanceName();
+ int pluginDefineId = instance.getPluginDefineId();
+ Optional<AlertChannel> alertChannelOptional =
alertPluginManager.getAlertChannel(instance.getPluginDefineId());
+ if (!alertChannelOptional.isPresent()) {
+ String message = String.format("Alert Plugin %s send error: the
channel doesn't exist, pluginDefineId: %s",
+ pluginInstanceName,
+ pluginDefineId);
+ logger.error("Alert Plugin {} send error : not found plugin {}",
pluginInstanceName, pluginDefineId);
+ return new AlertResult("false", message);
+ }
+ AlertChannel alertChannel = alertChannelOptional.get();
+
+ Map<String, String> paramsMap =
JSONUtils.toMap(instance.getPluginInstanceParams());
+ String instanceWarnType = WarningType.ALL.getDescp();
+
+ if (paramsMap != null) {
+ instanceWarnType =
paramsMap.getOrDefault(AlertConstants.NAME_WARNING_TYPE,
WarningType.ALL.getDescp());
+ }
+
+ WarningType warningType = WarningType.of(instanceWarnType);
+
+ if (warningType == null) {
+ String message = String.format("Alert Plugin %s send error : plugin
warnType is null", pluginInstanceName);
+ logger.error("Alert Plugin {} send error : plugin warnType is null",
pluginInstanceName);
+ return new AlertResult("false", message);
+ }
+
+ boolean sendWarning = false;
+ switch (warningType) {
+ case ALL:
+ sendWarning = true;
+ break;
+ case SUCCESS:
+ if (alertData.getWarnType() == WarningType.SUCCESS.getCode()) {
+ sendWarning = true;
+ }
+ break;
+ case FAILURE:
+ if (alertData.getWarnType() == WarningType.FAILURE.getCode()) {
+ sendWarning = true;
+ }
+ break;
+ default:
+ }
+
+ if (!sendWarning) {
+ logger.info(
+ "Alert Plugin {} send ignore warning type not match: plugin
warning type is {}, alert data warning type is {}",
+ pluginInstanceName, warningType.getCode(),
alertData.getWarnType());
+ return null;
+ }
+
+ AlertInfo alertInfo = AlertInfo.builder()
+ .alertData(alertData)
+ .alertParams(paramsMap)
+ .alertPluginInstanceId(instance.getId())
+ .build();
+ int waitTimeout = alertConfig.getWaitTimeout();
+ try {
+ AlertResult alertResult;
+ if (waitTimeout <= 0) {
+ if (alertData.getAlertType() == AlertType.CLOSE_ALERT.getCode()) {
+ alertResult = alertChannel.closeAlert(alertInfo);
+ } else {
+ alertResult = alertChannel.process(alertInfo);
+ }
+ } else {
+ CompletableFuture<AlertResult> future;
+ if (alertData.getAlertType() == AlertType.CLOSE_ALERT.getCode()) {
+ future = CompletableFuture.supplyAsync(() ->
alertChannel.closeAlert(alertInfo));
+ } else {
+ future = CompletableFuture.supplyAsync(() ->
alertChannel.process(alertInfo));
+ }
+ alertResult = future.get(waitTimeout, TimeUnit.MILLISECONDS);
+ }
+ if (alertResult == null) {
+ throw new RuntimeException("Alert result cannot be null");
+ }
+ return alertResult;
+ } catch (InterruptedException e) {
+ logger.error("send alert error alert data id :{},", alertData.getId(),
e);
+ Thread.currentThread().interrupt();
+ return new AlertResult("false", e.getMessage());
+ } catch (Exception e) {
+ logger.error("send alert error alert data id :{},", alertData.getId(),
e);
+ return new AlertResult("false", e.getMessage());
+ }
+}
+```
+
+综上描述，可以画出注册插件和发送消息的时序图
+
+
+
+以上就是告警插件的主要实现代码,是不是发现源码看下来也没有发现多高深和复杂：）所以多看看源码吧,以后你也可以写出这样优秀的开源软件来贡献开源
+
+参考连接
+>[[Feature] Alert Plugin Design · Issue #3049 · apache/dolphinscheduler
(github.com)](https://github.com/apache/dolphinscheduler/issues/3049)
+
+>[alert
(apache.org)](https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/contribute/backend/spi/alert.html)
\ No newline at end of file
diff --git
a/blog/zh-cn/Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md
b/blog/zh-cn/Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md
new file mode 100644
index 0000000000..161363c0b6
--- /dev/null
+++
b/blog/zh-cn/Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2.md
@@ -0,0 +1,276 @@
+[Apache
DolphinScheduler](https://dolphinscheduler.apache.org/zh-cn/docs/3.1.2/guide/upgrade/upgrade)
+**查看官方的升级文档，可知有提供升级脚本，如果只是跨小版本的更新那么只用执行脚本就好了，但跨多个大版本升级时依然容易出现各种问题，特此总结。**
+
+旧版本：1.3.4
+新版本：3.1.2
+
+## 1. 升级完成后使用资源中心报错 IllegalArgumentException: Failed to specify server's
Kerberos principal name
+资源中心使用的HDFS，开启了kerberos认证
+
+#### 解决方法：
+编辑 dolphinscheduler/api-server/conf/hdfs-site.xml 添加以下内容
+```xml
+<property>
+ <name>dfs.namenode.kerberos.principal.pattern</name>
+ <value>*</value>
+</property>
+```
+## 2. 升级完成后查看任务实例的日志，报错未找到日志
+**查看报错信息，检查新版本的目录结构和表里的日志路径，发现原因是新版本的日志路径有变更**
+**升级前的日志路径在 /logs/ 下**
+**升级后的日志路径在 /worker-server/logs/ 下**
+**因此需要修改这里的目录**
+#### 解决方法：
+执行sql修改日志路径
+```sql
+update t_ds_task_instance set
log_path=replace(log_path,'/logs/','/worker-server/logs/');
+```
+然后将原日志文件copy到新的日志路径
+```sql
+cp -r {旧版本dolphinscheduler目录}/logs/[1-9]*
{新版本dolphinscheduler目录}/worker-server/logs/*
+```
+## 3.升级完成后创建工作流报错
+查看报错信息，原因是 t_ds_process_definition_log 和 t_ds_process_definition 主键的初始值不一致
+那么修改成一致的就好
+#### 解决方法：
+执行sql
+```sql
+# 查出主键自增值
+select AUTO_INCREMENT FROM information_schema.TABLES WHERE TABLE_SCHEMA =
'dolphinscheduler' AND TABLE_NAME = 't_ds_process_definition' limit 1
+# 将上面sql的执行结果填写到下方参数处执行
+alter table dolphinscheduler_bak1.t_ds_process_definition_log auto_increment =
{max_id};
+```
+## 4.升级后任务实例列表为空
+检查查询的sql
+在`dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapper.xml`文件里，select
id="queryTaskInstanceListPaging"的sql
+```sql
+ select
+ <include refid="baseSqlV2">
+ <property name="alias" value="instance"/>
+ </include>
+ ,
+ process.name as process_instance_name
+ from t_ds_task_instance instance
+ left join t_ds_task_definition_log define on
define.code=instance.task_code and
define.version=instance.task_definition_version
+ left join t_ds_process_instance process on
process.id=instance.process_instance_id
+ where define.project_code = #{projectCode}
+ <if test="startTime != null">
+ and instance.start_time <![CDATA[ >=]]> #{startTime}
+ </if>
+ ......省略多余部分
+```
+**查询任务实例列表的sql会关联 t_ds_task_definition_log 表，经检查发现是
define.code=instance.task_code 这一句关联不上。
+结合下面的查询条件 define.project_code = #{projectCode} 可知，关联t_ds_task_definition_log
主要是为了过滤 projectCode，那么来修改下这个sql**
+#### 解决方法：
+```sql
+ select
+ <include refid="baseSqlV2">
+ <property name="alias" value="instance"/>
+ </include>
+ ,
+ process.name as process_instance_name
+ from t_ds_task_instance instance
+-- left join t_ds_task_definition_log define
+-- on define.code=instance.task_code and
+--
define.version=instance.task_definition_version
+ join t_ds_process_instance process
+ on process.id=instance.process_instance_id
+ join t_ds_process_definition define
+ on define.code=process.process_definition_code
+ where define.project_code = #{projectCode}
+ <if test="startTime != null">
+ and instance.start_time <![CDATA[ >=]]> #{startTime}
+ </if>
+ ......省略多余部分
+```
+**直接用 t_ds_process_definition 关联，也有project_code字段可以用来关联过滤**
+**这里修改后就能查出数据了**
+
+## 5. 执行升级脚本的过程中报错空指针
+### 5.1分析日志，定位到 UpgradeDao.java 517行
+查看代码
+```java
+513 if (TASK_TYPE_SUB_PROCESS.equals(taskType)) {
+514 JsonNode jsonNodeDefinitionId =
param.get("processDefinitionId");
+515 if (jsonNodeDefinitionId != null) {
+516 param.put("processDefinitionCode",
+517
processDefinitionMap.get(jsonNodeDefinitionId.asInt()).getCode());
+518 param.remove("processDefinitionId");
+519 }
+520 }
+```
+**很明显是** `processDefinitionMap.get(jsonNodeDefinitionId.asInt())`**
返回了null,加个null判断，如果返回null直接跳过，并将相关信息打印出来，升级结束后可以根据日志核对。**
+#### 解决方法：
+修改后
+```java
+if (jsonNodeDefinitionId != null) {
+ if (processDefinitionMap.get(jsonNodeDefinitionId.asInt()) != null) {
+
param.put("processDefinitionCode",processDefinitionMap.get(jsonNodeDefinitionId.asInt()).getCode());
+ param.remove("processDefinitionId");
+ } else {
+ logger.error("*******************error");
+ logger.error("*******************param:" + param);
+ logger.error("*******************jsonNodeDefinitionId:" +
jsonNodeDefinitionId);
+ }
+}
+```
+
+### 5.2分析日志，定位到 UpgradeDao.java 675行
+查看代码
+```java
+669 if (mapEntry.isPresent()) {
+670 Map.Entry<Long, Map<String, Long>>
processCodeTaskNameCodeEntry = mapEntry.get();
+671 dependItem.put("definitionCode",
processCodeTaskNameCodeEntry.getKey());
+672 String depTasks =
dependItem.get("depTasks").asText();
+673 long taskCode =
+674 "ALL".equals(depTasks) ||
processCodeTaskNameCodeEntry.getValue() == null ? 0L
+675 :
processCodeTaskNameCodeEntry.getValue().get(depTasks);
+676 dependItem.put("depTaskCode", taskCode);
+677 }
+```
+很明显是**processCodeTaskNameCodeEntry.getValue().get(depTasks)** 返回了null.
+修改下逻辑，不为null才赋值并打印相关日志
+#### 解决方法：
+修改后
+```java
+long taskCode =0;
+ if (processCodeTaskNameCodeEntry.getValue() != null
+
&&processCodeTaskNameCodeEntry.getValue().get(depTasks)!=null){
+ taskCode
=processCodeTaskNameCodeEntry.getValue().get(depTasks);
+ }else{
+ logger.error("********************
depTasks:"+depTasks);
+ logger.error("******************** taskCode
not in "+JSONUtils.toJsonString(processCodeTaskNameCodeEntry));
+ }
+ dependItem.put("depTaskCode", taskCode);
+```
+## 6.接入LDAP后登陆失败，不知道email字段名
+可在 api-server/conf/application.yaml 配置接入LDAP
+```yaml
+security:
+ authentication:
+ # Authentication types (supported types: PASSWORD,LDAP)
+ type: LDAP
+ # IF you set type `LDAP`, below config will be effective
+ ldap:
+ # ldap server config
+ urls: xxx
+ base-dn: xxx
+ username: xxx
+ password: xxx
+ user:
+ # admin userId when you use LDAP login
+ admin: xxx
+ identity-attribute: xxx
+ email-attribute: xxx
+ # action when ldap user is not exist (supported types: CREATE,DENY)
+ not-exist-action: CREATE
+```
+**要成功接入LDAP至少需要urls,base-dn,username,password,identity和email
正确填写，不知道email字段名可以按下面的方式处理，email先空着**
+**启动服务后用LDAP用户登录**
+#### 解决办法：
+** LDAP 认证的代码在
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/security/impl/ldap/LdapService.java
的 ldapLogin()**
+
+```java
+ctx = new InitialLdapContext(searchEnv, null);
+SearchControls sc = new SearchControls();
+sc.setReturningAttributes(new String[]{ldapEmailAttribute});
+sc.setSearchScope(SearchControls.SUBTREE_SCOPE);
+EqualsFilter filter = new EqualsFilter(ldapUserIdentifyingAttribute, userId);
+NamingEnumeration<SearchResult> results = ctx.search(ldapBaseDn,
filter.toString(), sc);
+if (results.hasMore()) {
+ // get the users DN (distinguishedName) from the result
+ SearchResult result = results.next();
+ NamingEnumeration<? extends Attribute> attrs =
result.getAttributes().getAll();
+ while (attrs.hasMore()) {
+ // Open another connection to the LDAP server with the found DN and
the password
+ searchEnv.put(Context.SECURITY_PRINCIPAL, result.getNameInNamespace());
+ searchEnv.put(Context.SECURITY_CREDENTIALS, userPwd);
+ try {
+ new InitialDirContext(searchEnv);
+ } catch (Exception e) {
+ logger.warn("invalid ldap credentials or ldap search error", e);
+ return null;
+ }
+ Attribute attr = attrs.next();
+ if (attr.getID().equals(ldapEmailAttribute)) {
+ return (String) attr.get();
+ }
+ }
+}
+```
+
+第三行会根据填的字段过滤，先注释第三行
+```java
+// sc.setReturningAttributes(new String[]{ldapEmailAttribute});
+```
+重新执行后第10行会返回全部字段
+```java
+NamingEnumeration<? extends Attribute> attrs = result.getAttributes().getAll();
+```
+通过打印或调试在里面找到email字段填到配置文件里，再还原上面注释的代码，重启服务后即可正常接入LDAP登录。
+## 7.管理员给普通用户授权资源文件不生效
+**经多次测试,发现普通用户只能看到所属用户为自己的资源文件，管理员授权后依然无法查看资源文件**
+
+#### 解决办法:
+**文件
`dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/permission/ResourcePermissionCheckServiceImpl.java`的`listAuthorizedResource()`方法,将
return 的集合修改为 relationResources**
+
+```java
+@Override
+ public Set<Integer> listAuthorizedResource(int userId, Logger logger) {
+ List<Resource> relationResources;
+ if (userId == 0) {
+ relationResources = new ArrayList<>();
+ } else {
+ // query resource relation
+ List<Integer> resIds =
resourceUserMapper.queryResourcesIdListByUserIdAndPerm(userId, 0);
+ relationResources = CollectionUtils.isEmpty(resIds) ? new
ArrayList<>() : resourceMapper.queryResourceListById(resIds);
+ }
+ List<Resource> ownResourceList =
resourceMapper.queryResourceListAuthored(userId, -1);
+ relationResources.addAll(ownResourceList);
+ return
relationResources.stream().map(Resource::getId).collect(toSet()); //
解决资源文件授权无效的问题
+// return
ownResourceList.stream().map(Resource::getId).collect(toSet());
+ }
+```
+检查新版本的change log ，发现在3.1.3版本修复了这个bug
+https://github.com/apache/dolphinscheduler/pull/13318
+## 8.kerberos过期的问题
+因为kerberos配置了票据过期时间，一段时间后资源中心的hdfs资源将无法访问，最好的解决办法是添加定时更新凭证的相关逻辑
+#### 解决办法:
+在文件
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/utils/CommonUtils.java
添加方法
+```java
+ /**
+ * * 定时更新凭证
+ */
+ private static void startCheckKeytabTgtAndReloginJob() {
+ // 每天循环，定时更新凭证
+ Executors.newScheduledThreadPool(1).scheduleWithFixedDelay(() -> {
+ try {
+
UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();
+ logger.warn("Check Kerberos Tgt And Relogin From Keytab
Finish.");
+ } catch (IOException e) {
+ logger.error("Check Kerberos Tgt And Relogin From Keytab
Error", e);
+ }
+ }, 0, 1, TimeUnit.DAYS);
+ logger.info("Start Check Keytab TGT And Relogin Job Success.");
+ }
+```
+然后在该文件的`loadKerberosConf` 方法返回 true 前调用
+```java
+public static boolean loadKerberosConf(String javaSecurityKrb5Conf, String
loginUserKeytabUsername,
+ String loginUserKeytabPath,
Configuration configuration) throws IOException {
+ if (CommonUtils.getKerberosStartupState()) {
+ System.setProperty(Constants.JAVA_SECURITY_KRB5_CONF,
StringUtils.defaultIfBlank(javaSecurityKrb5Conf,
+
PropertyUtils.getString(Constants.JAVA_SECURITY_KRB5_CONF_PATH)));
+ configuration.set(Constants.HADOOP_SECURITY_AUTHENTICATION,
Constants.KERBEROS);
+ UserGroupInformation.setConfiguration(configuration);
+ UserGroupInformation.loginUserFromKeytab(
+ StringUtils.defaultIfBlank(loginUserKeytabUsername,
+
PropertyUtils.getString(Constants.LOGIN_USER_KEY_TAB_USERNAME)),
+ StringUtils.defaultIfBlank(loginUserKeytabPath,
+
PropertyUtils.getString(Constants.LOGIN_USER_KEY_TAB_PATH)));
+ startCheckKeytabTgtAndReloginJob(); // 此处调用
+ return true;
+ }
+ return false;
+ }
+```
\ No newline at end of file
diff --git a/config/blog/en-us/tech.json b/config/blog/en-us/tech.json
index 177c8b8aa2..c620cb5690 100644
--- a/config/blog/en-us/tech.json
+++ b/config/blog/en-us/tech.json
@@ -54,5 +54,11 @@
"translator": "QuakeWang",
"dateStr": "2021-05-06",
"desc": "DAG: Full name Directed Acyclic Graph，referred to as DAG。Tasks in
the workflow are assembled in the form of directed acyclic graphs, which are
topologically traversed from nodes with zero indegrees of ingress until there
are no successor nodes."
+ },
+
"Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2":
{
+ "title": "Issue documentation and resolution of the Apache
DolphinScheduler upgrade from 1.3.4 to 3.1.2",
+ "author": "pinkfloyds",
+ "dateStr": "2024-01-19",
+ "desc": "By referring to the official upgrade documentation, it is known
that upgrade scripts are provided. If it is a minor version upgrade..."
}
}
diff --git a/config/blog/en-us/tutorial.json b/config/blog/en-us/tutorial.json
index fe9cdec800..9c69dd272f 100644
--- a/config/blog/en-us/tutorial.json
+++ b/config/blog/en-us/tutorial.json
@@ -28,5 +28,11 @@
"author": "LidongDai",
"dateStr": "2021-03-20",
"desc": "Apache Dolphin Scheduler(Incubating) will organize a meetup in
Shanghai 2019.10.26. Welcome to register."
+ },
+
"How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications":
{
+ "title": "How to use Apache DolphinScheduler for targeted alarm plugin
development for enterprise applications?",
+ "author": "pinkfloyds",
+ "dateStr": "2024-01-19",
+ "desc": "In the 2.0.1 version of Apache DolphinScheduler, a plugin
architecture improvement was introduced to enhance the flexibility..."
}
}
diff --git a/config/blog/zh-cn/tech.json b/config/blog/zh-cn/tech.json
index 887965f4ca..17a969b924 100644
--- a/config/blog/zh-cn/tech.json
+++ b/config/blog/zh-cn/tech.json
@@ -64,5 +64,11 @@
"target": "_blank",
"desc": "这里推荐一下社区贡献者 import 的实践文章，写的非常赞，sudo 也是非常的痛点，也寻求一种更好的实现方式来实现多租户",
"link": "https://mp.weixin.qq.com/s/5rRWMoT0DLMcOdDl-mrsRQ";
+ },
+
"Issue_documentation_and_resolution_of_the_Apache_DolphinScheduler_upgrade_from1.3.4_to_3.1.2":
{
+ "title": "Apache DolphinScheduler 从 1.3.4 升级至3.1.2 过程中的问题记录及解决方案",
+ "author": "刘宇星",
+ "dateStr": "2024-01-19",
+ "desc": "Apache
DolphinScheduler官方的升级文档提供了升级脚本，如果只是跨小版本的更新，那么只用执行脚本就好了，但跨多个大版本升级时依然容易出现各种问题，特此总结..."
}
}
diff --git a/config/blog/zh-cn/tutorial.json b/config/blog/zh-cn/tutorial.json
index 4bdad745e1..c0dc96e126 100644
--- a/config/blog/zh-cn/tutorial.json
+++ b/config/blog/zh-cn/tutorial.json
@@ -35,5 +35,11 @@
"dateStr": "2021-01-05",
"desc": "本文档为dolphinscheduler配置文件指南,针对版本为 dolphinscheduler-1.3.x 版本",
"link": "https://mp.weixin.qq.com/s/gJUsGBHuMxw79VGH2ziZtg";
+ },
+
"How_to_use_Apache_DolphinScheduler_for_targeted_alarm_plugin_development_for_enterprise_applications":
{
+ "title": "企业级应用如何用 Apache DolphinScheduler 有针对性地进行告警插件开发？",
+ "author": "刘宇星",
+ "dateStr": "2024-01-19",
+ "desc": "Apache
DolphinScheduler的2.0.1版本加入了插件化架构改进，将任务、告警组件、数据源、资源存储、注册中心等都将被设计为扩展点，..."
}
}
