Re: [I] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
qingwli commented on issue #15423: URL: https://github.com/apache/dolphinscheduler/issues/15423#issuecomment-2030950130 > > Compatibility, Deprecation, and Migration Plan > > > Compatible with current system, aspect all api method, but just create, delete, method will into record log logic, other query method will keep same. Just record log not change anything about api request. > > It seems is not compatible with current system, since you need to change the schema in database. Yes, I changed the schema, but this table is never used before, just defined with no data -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
qingwli commented on code in PR #15554:
URL:
https://github.com/apache/dolphinscheduler/pull/15554#discussion_r1547058643
##
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/AuditObjectType.java:
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.common.enums;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
+import lombok.Getter;
+
+/**
+ * Audit Object type
+ */
+@Getter
+public enum AuditObjectType {
+
+PROJECT("Project", null), // 1
+PROCESS("Process", PROJECT),
+PROCESS_INSTANCE("Process instance", PROCESS),
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Directly use Quartz replase api to bootstrap a schedule [dolphinscheduler]
sonarcloud[bot] commented on PR #15781: URL: https://github.com/apache/dolphinscheduler/pull/15781#issuecomment-2031126210 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15781) **Quality Gate failed** Failed conditions  [30.2% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15781=new_coverage=list) (required ≥ 60%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15781) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Directly use Quartz replase api to bootstrap a schedule [dolphinscheduler]
sonarcloud[bot] commented on PR #15781: URL: https://github.com/apache/dolphinscheduler/pull/15781#issuecomment-2031127148 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15781) **Quality Gate failed** Failed conditions  [30.2% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15781=new_coverage=list) (required ≥ 60%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15781) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Improvement-15744][parameter] project parameter add update time and update user id [dolphinscheduler]
sonarcloud[bot] commented on PR #15745: URL: https://github.com/apache/dolphinscheduler/pull/15745#issuecomment-2030922287 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15745) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15745=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15745=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15745=false=true)  [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15745=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15745=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15745) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Improvement-15744][parameter] project parameter add update time and update user id [dolphinscheduler]
sonarcloud[bot] commented on PR #15745: URL: https://github.com/apache/dolphinscheduler/pull/15745#issuecomment-2030922132 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15745) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15745=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15745=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15745=false=true)  [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15745=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15745=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15745) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement][parameter] When modifying project parameters, it is not possible to identify who modified the relevant person and time [dolphinscheduler]
SbloodyS closed issue #15744: [Improvement][parameter] When modifying project parameters, it is not possible to identify who modified the relevant person and time URL: https://github.com/apache/dolphinscheduler/issues/15744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Improvement-15744][parameter] project parameter add update time and update user id [dolphinscheduler]
SbloodyS merged PR #15745: URL: https://github.com/apache/dolphinscheduler/pull/15745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
qingwli commented on code in PR #15554:
URL:
https://github.com/apache/dolphinscheduler/pull/15554#discussion_r1547060184
##
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/operator/Operator.java:
##
@@ -15,14 +15,13 @@
* limitations under the License.
*/
-package org.apache.dolphinscheduler.api.audit;
+package org.apache.dolphinscheduler.api.audit.operator;
-public interface AuditSubscriber {
+import org.apache.dolphinscheduler.api.audit.enums.AuditType;
-/**
- * process the audit message
- *
- * @param message
- */
-void execute(AuditMessage message);
+import org.aspectj.lang.ProceedingJoinPoint;
+
+public interface Operator {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
qingwli commented on code in PR #15554:
URL:
https://github.com/apache/dolphinscheduler/pull/15554#discussion_r1547068414
##
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##
@@ -293,7 +293,7 @@ private void checkMasterExists() {
// no master
if (masterServers.isEmpty()) {
-throw new ServiceException(Status.MASTER_NOT_EXISTS);
+// throw new ServiceException(Status.MASTER_NOT_EXISTS);
Review Comment:
done
##
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/operator/BaseOperator.java:
##
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.audit.operator;
+
+import org.apache.dolphinscheduler.api.audit.OperatorUtils;
+import org.apache.dolphinscheduler.api.audit.enums.AuditType;
+import org.apache.dolphinscheduler.api.service.AuditService;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.dao.entity.AuditLog;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+import java.util.Map;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.aspectj.lang.ProceedingJoinPoint;
+import org.aspectj.lang.reflect.MethodSignature;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import com.google.common.base.Strings;
+
+@Service
+@Slf4j
+public abstract class BaseOperator implements Operator {
+
+@Autowired
+private AuditService auditService;
+
+@Override
+public Object recordAudit(ProceedingJoinPoint point, String describe,
AuditType auditType) throws Throwable {
+long beginTime = System.currentTimeMillis();
+
+MethodSignature signature = (MethodSignature) point.getSignature();
+Map paramsMap = OperatorUtils.getParamsMap(point,
signature);
+
+User user = OperatorUtils.getUser(paramsMap);
+
+if (user == null) {
+log.error("user is null");
+return point.proceed();
+}
+
+List auditLogList =
OperatorUtils.buildAuditLogList(describe, auditType, user);
+setRequestParam(auditType, auditLogList, paramsMap);
+
+Result result = (Result) point.proceed();
+if (OperatorUtils.resultFail(result)) {
+log.error("request fail, code {}", result.getCode());
+return result;
+}
+
+setObjectIdentityFromReturnObject(auditType, result, auditLogList);
+
+modifyAuditOperationType(auditType, paramsMap, auditLogList);
+modifyAuditObjectType(auditType, paramsMap, auditLogList);
+
+long latency = System.currentTimeMillis() - beginTime;
+auditService.addAudit(auditLogList, latency);
+
+return result;
+}
+
+protected void setRequestParam(AuditType auditType, List
auditLogList, Map paramsMap) {
+String[] paramNameArr = auditType.getRequestParamName();
+
+if (paramNameArr.length == 0) {
+return;
+}
+
+modifyRequestParams(paramNameArr, paramsMap, auditLogList);
+setObjectByParma(paramNameArr, paramsMap, auditLogList);
+
+if (auditLogList.get(0).getObjectId() == null) {
+
auditLogList.get(0).setObjectId(OperatorUtils.getObjectIdentityByParma(paramNameArr,
paramsMap));
+}
+}
+
+protected void setObjectByParma(String[] paramNameArr, Map
paramsMap,
+List auditLogList) {
+
+String name = paramNameArr[0];
+Object value = paramsMap.get(name);
+
+if (value == null) {
+return;
+}
+
+String objName = getObjectNameFromReturnIdentity(value);
+
+if (Strings.isNullOrEmpty(objName)) {
+auditLogList.get(0).setObjectName(value.toString());
+return;
+}
+
+try {
+long objectId = Long.parseLong(value.toString());
+auditLogList.get(0).setObjectId(objectId);
+} catch (NumberFormatException e) {
+log.error("value is not long, value: {}", value);
+}
+
+
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
sonarcloud[bot] commented on PR #15554: URL: https://github.com/apache/dolphinscheduler/pull/15554#issuecomment-2031016120 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) **Quality Gate passed** Issues  [34 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15554=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15554=false=true)  [62.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
sonarcloud[bot] commented on PR #15554: URL: https://github.com/apache/dolphinscheduler/pull/15554#issuecomment-2031017326 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) **Quality Gate passed** Issues  [34 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15554=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15554=false=true)  [62.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
qingwli commented on code in PR #15554: URL: https://github.com/apache/dolphinscheduler/pull/15554#discussion_r1547057095 ## dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql: ## @@ -2008,10 +2008,14 @@ DROP TABLE IF EXISTS `t_ds_audit_log`; CREATE TABLE `t_ds_audit_log` ( `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT'key', `user_id` int(11) NOT NULL COMMENT 'user id', - `resource_type` int(11) NOT NULL COMMENT 'resource type', - `operation` int(11) NOT NULL COMMENT 'operation', - `time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', - `resource_id` int(11) NULL DEFAULT NULL COMMENT 'resource id', + `object_id` bigint(20) DEFAULT NULL COMMENT 'object id', + `object_name` varchar(100) DEFAULT NULL COMMENT 'object id', + `object_type` varchar(100) NOT NULL COMMENT 'object type', Review Comment: For this part, I defined object id means the object I modified, `task instance` it's not a module. `File `is not a domain, I think `object` is more suit for this, can cover all the stuff we record, WDYT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
github-advanced-security[bot] commented on code in PR #15554:
URL:
https://github.com/apache/dolphinscheduler/pull/15554#discussion_r1547095110
##
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/operator/BaseAuditOperator.java:
##
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.audit.operator;
+
+import org.apache.dolphinscheduler.api.audit.OperatorUtils;
+import org.apache.dolphinscheduler.api.audit.enums.AuditType;
+import org.apache.dolphinscheduler.api.service.AuditService;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.dao.entity.AuditLog;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import org.apache.commons.lang.math.NumberUtils;
+
+import java.util.List;
+import java.util.Map;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.aspectj.lang.ProceedingJoinPoint;
+import org.aspectj.lang.reflect.MethodSignature;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import com.google.common.base.Strings;
+
+@Service
+@Slf4j
+public abstract class BaseAuditOperator implements AuditOperator {
+
+@Autowired
+private AuditService auditService;
+
+@Override
+public Object recordAudit(ProceedingJoinPoint point, String describe,
AuditType auditType) throws Throwable {
+long beginTime = System.currentTimeMillis();
+
+MethodSignature signature = (MethodSignature) point.getSignature();
+Map paramsMap = OperatorUtils.getParamsMap(point,
signature);
+
+User user = OperatorUtils.getUser(paramsMap);
+
+if (user == null) {
+log.error("user is null");
+return point.proceed();
+}
+
+List auditLogList =
OperatorUtils.buildAuditLogList(describe, auditType, user);
+setRequestParam(auditType, auditLogList, paramsMap);
+
+Result result = (Result) point.proceed();
+if (OperatorUtils.resultFail(result)) {
+log.error("request fail, code {}", result.getCode());
+return result;
+}
+
+setObjectIdentityFromReturnObject(auditType, result, auditLogList);
+
+modifyAuditOperationType(auditType, paramsMap, auditLogList);
+modifyAuditObjectType(auditType, paramsMap, auditLogList);
+
+long latency = System.currentTimeMillis() - beginTime;
+auditService.addAudit(auditLogList, latency);
+
+return result;
+}
+
+protected void setRequestParam(AuditType auditType, List
auditLogList, Map paramsMap) {
+String[] paramNameArr = auditType.getRequestParamName();
+
+if (paramNameArr.length == 0) {
+return;
+}
+
+modifyRequestParams(paramNameArr, paramsMap, auditLogList);
+setObjectByParma(paramNameArr, paramsMap, auditLogList);
+
+if (auditLogList.get(0).getObjectId() == null) {
+
auditLogList.get(0).setObjectId(OperatorUtils.getObjectIdentityByParma(paramNameArr,
paramsMap));
+}
+}
+
+protected void setObjectByParma(String[] paramNameArr, Map
paramsMap,
+List auditLogList) {
+
+String name = paramNameArr[0];
+Object value = paramsMap.get(name);
+
+if (value == null) {
+return;
+}
+
+String objName = getObjectNameFromReturnIdentity(value);
+
+if (Strings.isNullOrEmpty(objName)) {
+auditLogList.get(0).setObjectName(value.toString());
+return;
+}
+
+try {
+long objectId = Long.parseLong(value.toString());
+auditLogList.get(0).setObjectId(objectId);
+} catch (NumberFormatException e) {
+log.error("value is not long, value: {}", value);
+}
+
+auditLogList.get(0).setObjectName(objName);
+}
+
+protected void setObjectByParmaArr(String[] paramNameArr, Map paramsMap,
+ List auditLogList) {
+
+AuditLog auditLog = auditLogList.get(0);
+for (String param : paramNameArr) {
+if (!paramsMap.containsKey(param)) {
+
(dolphinscheduler-website) branch asf-site updated: Automated deployment: 0980f3b270a1b9fed9d4efe5053c8d1c6c4ad553
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 8ea05e220a Automated deployment: 0980f3b270a1b9fed9d4efe5053c8d1c6c4ad553 8ea05e220a is described below commit 8ea05e220a715d83adee49f9a0afcf1709158a59 Author: github-actions[bot] AuthorDate: Tue Apr 2 05:05:55 2024 + Automated deployment: 0980f3b270a1b9fed9d4efe5053c8d1c6c4ad553 --- python/2.0.5/.buildinfo | 2 +- python/2.0.6/.buildinfo | 2 +- python/2.0.7/.buildinfo | 2 +- python/3.0.0/.buildinfo | 2 +- python/3.0.1/.buildinfo | 2 +- python/3.1.0/.buildinfo | 2 +- python/4.0.0/.buildinfo | 2 +- python/4.0.1/.buildinfo | 2 +- python/4.0.2/.buildinfo | 2 +- python/4.0.3/.buildinfo | 2 +- python/4.0.4/.buildinfo | 2 +- python/main/.buildinfo | 2 +- 12 files changed, 12 insertions(+), 12 deletions(-) diff --git a/python/2.0.5/.buildinfo b/python/2.0.5/.buildinfo index 16b94e3c17..c3bf6c468c 100644 --- a/python/2.0.5/.buildinfo +++ b/python/2.0.5/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: d13181629b17dd5f58283f3c2eaffcfe +config: d979dc6bcf16ed61d3ca2007568cf6e9 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/2.0.6/.buildinfo b/python/2.0.6/.buildinfo index 231c738f62..a6243f6a4a 100644 --- a/python/2.0.6/.buildinfo +++ b/python/2.0.6/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: ebf3cf2d50cae6f105459ea18122fb7b +config: cb558631379a089e99f996175ec1 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/2.0.7/.buildinfo b/python/2.0.7/.buildinfo index d726385aa3..179da5423d 100644 --- a/python/2.0.7/.buildinfo +++ b/python/2.0.7/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 42966ad78f4206034df43943c480a039 +config: 388cd96b7e9e3953f2b5d37a44acbbe2 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/3.0.0/.buildinfo b/python/3.0.0/.buildinfo index b993b18c67..2a54c9c689 100644 --- a/python/3.0.0/.buildinfo +++ b/python/3.0.0/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 16b447741ba76ab251e57e31495b6799 +config: 6f501d1537947a02ebbae777c88e19b1 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/3.0.1/.buildinfo b/python/3.0.1/.buildinfo index 351cca6681..e9dfaa728e 100644 --- a/python/3.0.1/.buildinfo +++ b/python/3.0.1/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 1148a1cb959e5e7df3321d92e6b0af6e +config: cb2c5c77a8ba735a5b916af0b63106d6 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/3.1.0/.buildinfo b/python/3.1.0/.buildinfo index 38d052e294..b21c47e889 100644 --- a/python/3.1.0/.buildinfo +++ b/python/3.1.0/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: ddcb38fad84e8dca1b70bba4078360b9 +config: e236f7454d35be91826ed1245cfc43e9 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/4.0.0/.buildinfo b/python/4.0.0/.buildinfo index dd6393b65e..47e43a0a18 100644 --- a/python/4.0.0/.buildinfo +++ b/python/4.0.0/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 55f27ca70d39bb720f3ce4222188fbe7 +config: 86a9be23a549eeebf98b5ec1dc73d705 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/4.0.1/.buildinfo b/python/4.0.1/.buildinfo index 1f25baa23e..3f84bc56d3 100644 --- a/python/4.0.1/.buildinfo +++ b/python/4.0.1/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 3ead1ca6f6c38fe501031cde388fffa6 +config: 6699c6a6517ca32b8db1526654008c4c tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/python/4.0.2/.buildinfo b/python/4.0.2/.buildinfo index 5e3ca13ea5..c3e8d64af2 100644 --- a/python/4.0.2/.buildinfo +++ b/python/4.0.2/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 6eabd3b519004e1b6ea6e761ae10
(dolphinscheduler) branch dev updated: [Improvement-15744][parameter] project parameter add update time and update user id (#15745)
This is an automated email from the ASF dual-hosted git repository.
zihaoxiang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
The following commit(s) were added to refs/heads/dev by this push:
new 920ac154cb [Improvement-15744][parameter] project parameter add update
time and update user id (#15745)
920ac154cb is described below
commit 920ac154cb9c0c8f33670af7c67387c0e30f6dd5
Author: 小可耐 <46134044+sdh...@users.noreply.github.com>
AuthorDate: Tue Apr 2 09:42:44 2024 +0800
[Improvement-15744][parameter] project parameter add update time and update
user id (#15745)
* project parameter add update time and update user id
* project parameter add comment operator user id and UT
* project parameter add ui
---
.../api/service/impl/ProjectParameterServiceImpl.java | 2 ++
.../api/service/ProjectParameterServiceTest.java | 3 +++
.../dolphinscheduler/dao/entity/ProjectParameter.java | 8
.../dolphinscheduler/dao/mapper/ProjectParameterMapper.xml | 14 +-
.../src/main/resources/sql/dolphinscheduler_h2.sql | 1 +
.../src/main/resources/sql/dolphinscheduler_mysql.sql | 1 +
.../src/main/resources/sql/dolphinscheduler_postgresql.sql | 1 +
.../upgrade/3.2.2_schema/mysql/dolphinscheduler_ddl.sql| 1 +
.../3.2.2_schema/postgresql/dolphinscheduler_ddl.sql | 4 +++-
dolphinscheduler-ui/src/locales/en_US/project.ts | 2 ++
dolphinscheduler-ui/src/locales/zh_CN/project.ts | 2 ++
.../src/views/projects/parameter/use-table.ts | 10 ++
12 files changed, 43 insertions(+), 6 deletions(-)
diff --git
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java
index 6e66f6286e..e30375e809 100644
---
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java
+++
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java
@@ -155,6 +155,8 @@ public class ProjectParameterServiceImpl extends
BaseServiceImpl implements Proj
projectParameter.setParamName(projectParameterName);
projectParameter.setParamValue(projectParameterValue);
+projectParameter.setUpdateTime(new Date());
+projectParameter.setOperator(loginUser.getId());
if (projectParameterMapper.updateById(projectParameter) > 0) {
log.info("Project parameter is updated and id is :{}",
projectParameter.getId());
diff --git
a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java
b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java
index 4ab22bda21..7a3fb1b68d 100644
---
a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java
+++
b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java
@@ -98,6 +98,9 @@ public class ProjectParameterServiceTest {
Mockito.when(projectParameterMapper.updateById(Mockito.any())).thenReturn(1);
result = projectParameterService.updateProjectParameter(loginUser,
projectCode, 1, "key1", "value");
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
+ProjectParameter projectParameter = (ProjectParameter)
result.getData();
+Assertions.assertNotNull(projectParameter.getOperator());
+Assertions.assertNotNull(projectParameter.getUpdateTime());
}
@Test
diff --git
a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java
b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java
index fbeeb387f1..03e9140145 100644
---
a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java
+++
b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java
@@ -42,6 +42,8 @@ public class ProjectParameter {
@TableField("user_id")
private Integer userId;
+private Integer operator;
+
private long code;
@TableField("project_code")
@@ -56,4 +58,10 @@ public class ProjectParameter {
private Date createTime;
private Date updateTime;
+
+@TableField(exist = false)
+private String createUser;
+
+@TableField(exist = false)
+private String modifyUser;
}
diff --git
a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml
b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml
index 159c263ea4..5b22d40a81 100644
---
Re: [I] The spark submitted task runs successfully, but the UI interface icon shows failure, ds version 1.3.4 [dolphinscheduler]
github-actions[bot] commented on issue #4981: URL: https://github.com/apache/dolphinscheduler/issues/4981#issuecomment-2030838995 This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] [API] queryAllProjectList response userName is null [dolphinscheduler]
github-actions[bot] commented on issue #6766: URL: https://github.com/apache/dolphinscheduler/issues/6766#issuecomment-2030838952 This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] The spark submitted task runs successfully, but the UI interface icon shows failure, ds version 1.3.4 [dolphinscheduler]
github-actions[bot] closed issue #4981: The spark submitted task runs successfully, but the UI interface icon shows failure, ds version 1.3.4 URL: https://github.com/apache/dolphinscheduler/issues/4981 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
qingwli commented on code in PR #15554:
URL:
https://github.com/apache/dolphinscheduler/pull/15554#discussion_r1547059530
##
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/OperatorLogAspect.java:
##
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.audit;
+
+import org.apache.dolphinscheduler.api.audit.operator.Operator;
+import org.apache.dolphinscheduler.service.bean.SpringApplicationContext;
+
+import java.lang.reflect.Method;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.aspectj.lang.ProceedingJoinPoint;
+import org.aspectj.lang.annotation.Around;
+import org.aspectj.lang.annotation.Aspect;
+import org.aspectj.lang.annotation.Pointcut;
+import org.aspectj.lang.reflect.MethodSignature;
+import org.springframework.stereotype.Component;
+
+import io.swagger.v3.oas.annotations.Operation;
+
+@Aspect
+@Slf4j
+@Component
+public class OperatorLogAspect {
+
+@Pointcut("@annotation(org.apache.dolphinscheduler.api.audit.OperatorLog)")
+public void logPointCut() {
+}
+
+@Around("logPointCut()")
+public Object around(ProceedingJoinPoint point) throws Throwable {
+MethodSignature signature = (MethodSignature) point.getSignature();
+Method method = signature.getMethod();
+
+OperatorLog operatorLog = method.getAnnotation(OperatorLog.class);
+// Api don't need record log
+if (operatorLog == null) {
+return point.proceed();
+}
+
+Operation operation = method.getAnnotation(Operation.class);
Review Comment:
```
@Operation(summary = "queryAuditObjectTypeList", description =
"QUERY_AUDIT_OBJECT_TYPE_LIST")
@GetMapping(value = "/audit-log-object-type")
@ResponseStatus(HttpStatus.OK)
@ApiException(QUERY_AUDIT_LOG_LIST_PAGING)
public Result> queryAuditObjectTypeList() {
return Result.success(AuditObjectTypeDto.getObjectTypeDtoList());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
sonarcloud[bot] commented on PR #15554: URL: https://github.com/apache/dolphinscheduler/pull/15554#issuecomment-2030969234 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) **Quality Gate passed** Issues  [42 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15554=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15554=false=true)  [62.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
qingwli commented on code in PR #15554:
URL:
https://github.com/apache/dolphinscheduler/pull/15554#discussion_r1547065411
##
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/operator/BaseOperator.java:
##
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.audit.operator;
+
+import org.apache.dolphinscheduler.api.audit.OperatorUtils;
+import org.apache.dolphinscheduler.api.audit.enums.AuditType;
+import org.apache.dolphinscheduler.api.service.AuditService;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.dao.entity.AuditLog;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+import java.util.Map;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.aspectj.lang.ProceedingJoinPoint;
+import org.aspectj.lang.reflect.MethodSignature;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import com.google.common.base.Strings;
+
+@Service
+@Slf4j
+public abstract class BaseOperator implements Operator {
+
+@Autowired
+private AuditService auditService;
+
+@Override
+public Object recordAudit(ProceedingJoinPoint point, String describe,
AuditType auditType) throws Throwable {
+long beginTime = System.currentTimeMillis();
+
+MethodSignature signature = (MethodSignature) point.getSignature();
+Map paramsMap = OperatorUtils.getParamsMap(point,
signature);
+
+User user = OperatorUtils.getUser(paramsMap);
+
+if (user == null) {
+log.error("user is null");
+return point.proceed();
+}
+
+List auditLogList =
OperatorUtils.buildAuditLogList(describe, auditType, user);
+setRequestParam(auditType, auditLogList, paramsMap);
+
+Result result = (Result) point.proceed();
+if (OperatorUtils.resultFail(result)) {
+log.error("request fail, code {}", result.getCode());
+return result;
+}
+
+setObjectIdentityFromReturnObject(auditType, result, auditLogList);
+
+modifyAuditOperationType(auditType, paramsMap, auditLogList);
+modifyAuditObjectType(auditType, paramsMap, auditLogList);
+
+long latency = System.currentTimeMillis() - beginTime;
+auditService.addAudit(auditLogList, latency);
+
+return result;
Review Comment:
todo https://github.com/apache/dolphinscheduler/issues/15788
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Improvement][Audit log] Record Api request and response when occur error [dolphinscheduler]
qingwli opened a new issue, #15788: URL: https://github.com/apache/dolphinscheduler/issues/15788 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement. ### Description Audit the case when proceeding throw exception. ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DSIP-26][Audit log] Audit log improvement design [dolphinscheduler]
sonarcloud[bot] commented on PR #15554: URL: https://github.com/apache/dolphinscheduler/pull/15554#issuecomment-2030970991 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) **Quality Gate passed** Issues  [42 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15554=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15554=false=true)  [62.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15554=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15554) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] sql任务报错，sql解析的不对 [dolphinscheduler]
xiamud opened a new issue, #15787: URL: https://github.com/apache/dolphinscheduler/issues/15787 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened SQL作业使用oracle数据源类型，sql中使用union all时任务出错。 执行以下sql： select a,b,c,b-c as d from (select 1 as a,2 as b,3 as c from dual) union all select 1 as a,2 as b,3 as c,4 as d from dual 通过查询日志，可以看到以上sql被解析为： SELECT a, b, c, b - c AS d FROM ( SELECT 1 AS a, 2 AS b, 3 AS c FROM dual UNION ALL SELECT 1 AS a, 2 AS b, 3 AS c, 4 AS d FROM dual ) ### What you expected to happen 正常显示合并结果集 ### How to reproduce 在SQL任务中执行以下sql： select a,b,c,b-c as d from (select 1 as a,2 as b,3 as c from dual) union all select 1 as a,2 as b,3 as c,4 as d from dual ### Anything else _No response_ ### Version 3.2.x ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] The sql task reports an error, and the sql parsing is incorrect. [dolphinscheduler]
github-actions[bot] commented on issue #15787: URL: https://github.com/apache/dolphinscheduler/issues/15787#issuecomment-2029427466 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened The SQL job uses the oracle data source type, and the task error occurs when union all is used in sql. Execute the following sql: select a,b,c,b-c as d from (select 1 as a,2 as b,3 as c from dual) union all select 1 as a,2 as b,3 as c,4 as d from dual By querying the log, you can see that the above sql is parsed as: SELECT a, b, c, b - c AS d FROM ( SELECT 1 AS a, 2 AS b, 3 AS c FROM dual UNION ALL SELECT 1 AS a, 2 AS b, 3 AS c, 4 AS d FROM dual ) ### What you expected to happen Display the merged result set normally ### How to reproduce Execute the following sql in the SQL task: select a,b,c,b-c as d from (select 1 as a,2 as b,3 as c from dual) union all select 1 as a,2 as b,3 as c,4 as d from dual ### Anything else _No response_ ### Version 3.2.x ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Improvement] add resource full name check [dolphinscheduler]
sonarcloud[bot] commented on PR #15786: URL: https://github.com/apache/dolphinscheduler/pull/15786#issuecomment-2029518839 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) **Quality Gate passed** Issues  [8 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15786=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15786=false=true)  [86.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Improvement] add resource full name check [dolphinscheduler]
caishunfeng merged PR #15786: URL: https://github.com/apache/dolphinscheduler/pull/15786 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(dolphinscheduler) branch dev updated: [Improvement] add resource full name check (#15786)
This is an automated email from the ASF dual-hosted git repository.
caishunfeng pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
The following commit(s) were added to refs/heads/dev by this push:
new 8acc697942 [Improvement] add resource full name check (#15786)
8acc697942 is described below
commit 8acc69794275149a359a711359873b39781f58b7
Author: caishunfeng
AuthorDate: Mon Apr 1 22:49:59 2024 +0800
[Improvement] add resource full name check (#15786)
* [Improvement] add resource full name check
---
.../apache/dolphinscheduler/api/enums/Status.java | 2 +
.../api/service/ResourcesService.java | 6 +-
.../api/service/impl/ResourcesServiceImpl.java | 20 +-
.../api/service/ResourcesServiceTest.java | 333 +
4 files changed, 168 insertions(+), 193 deletions(-)
diff --git
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
index 113ccb6bd1..9dc12d9ba0 100644
---
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
+++
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
@@ -323,6 +323,8 @@ public enum Status {
REMOVE_TASK_INSTANCE_CACHE_ERROR(20019, "remove task instance cache
error", "删除任务实例缓存错误"),
+ILLEGAL_RESOURCE_PATH(20020, "Resource file [{0}] is illegal",
"非法的资源路径[{0}]"),
+
USER_NO_OPERATION_PERM(30001, "user has no operation privilege",
"当前用户没有操作权限"),
USER_NO_OPERATION_PROJECT_PERM(30002, "user {0} is not has project {1}
permission", "当前用户[{0}]没有[{1}]项目的操作权限"),
USER_NO_WRITE_PROJECT_PERM(30003, "user [{0}] does not have write
permission for project [{1}]",
diff --git
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ResourcesService.java
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ResourcesService.java
index 24d1ba8727..54023baad9 100644
---
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ResourcesService.java
+++
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ResourcesService.java
@@ -194,13 +194,13 @@ public interface ResourcesService {
org.springframework.core.io.Resource downloadResource(User loginUser,
String fullName) throws IOException;
/**
- * Get resource by given resource type and full name.
+ * Get resource by given resource type and file name.
* Useful in Python API create task which need processDefinition
information.
*
* @param userName user who query resource
- * @param fullName full name of the resource
+ * @param fileName file name of the resource
*/
-StorageEntity queryFileStatus(String userName, String fullName) throws
Exception;
+StorageEntity queryFileStatus(String userName, String fileName) throws
Exception;
/**
* delete DATA_TRANSFER data in resource center
diff --git
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
index e6341f021d..6a15da17a8 100644
---
a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
+++
b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
@@ -126,6 +126,7 @@ public class ResourcesServiceImpl extends BaseServiceImpl
implements ResourcesSe
}
String tenantCode = getTenantCode(user);
+checkFullName(tenantCode, currentDir);
String userResRootPath = ResourceType.UDF.equals(type) ?
storageOperate.getUdfDir(tenantCode)
: storageOperate.getResDir(tenantCode);
@@ -171,6 +172,7 @@ public class ResourcesServiceImpl extends BaseServiceImpl
implements ResourcesSe
}
String tenantCode = getTenantCode(user);
+checkFullName(tenantCode, currentDir);
result = verifyFile(name, type, file);
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
@@ -257,6 +259,7 @@ public class ResourcesServiceImpl extends BaseServiceImpl
implements ResourcesSe
}
String tenantCode = getTenantCode(user);
+checkFullName(tenantCode, resourceFullName);
if (!isUserTenantValid(isAdmin(loginUser), tenantCode, resTenantCode))
{
log.error("current user does not have permission");
@@ -264,7 +267,7 @@ public class ResourcesServiceImpl extends BaseServiceImpl
implements ResourcesSe
return result;
}
-String defaultPath = storageOperate.getResDir(tenantCode);
+String defaultPath = storageOperate.getDir(type, tenantCode);
StorageEntity resource;
try {
@@ -949,6
Re: [PR] [Improvement] add resource full name check [dolphinscheduler]
sonarcloud[bot] commented on PR #15786: URL: https://github.com/apache/dolphinscheduler/pull/15786#issuecomment-2029455718 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) **Quality Gate passed** Issues  [8 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15786=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15786=false=true)  [94.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] [workflow] SQL type run as description [dolphinscheduler]
songwenyong commented on issue #15760: URL: https://github.com/apache/dolphinscheduler/issues/15760#issuecomment-2029497435 The bug occurs in the `DataSourceProcessor.splitAndRemoveComment` method, which calls the Alibaba Druid's `SQLParserUtils.splitAndRemoveComment` method. When processing SQL, there's an issue with the Lexer parser configuration, as follows: `lexer.config(SQLParserFeature.SkipComments, false);` `lexer.config(SQLParserFeature.KeepComments, true);` It is recommended to modify the `DataSourceProcessor.splitAndRemoveComment` method to make two calls. First, call `SQLParserUtils.removeComment`, and then call `SQLParserUtils.split`. We have already fixed this in our company's branch, and I can submit a PR if possible.  @davidzollo @SbloodyS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Improvement] add resource full name check [dolphinscheduler]
sonarcloud[bot] commented on PR #15786: URL: https://github.com/apache/dolphinscheduler/pull/15786#issuecomment-2029457394 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) **Quality Gate passed** Issues  [8 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15786=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15786=false=true)  [94.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Improvement] add resource full name check [dolphinscheduler]
sonarcloud[bot] commented on PR #15786: URL: https://github.com/apache/dolphinscheduler/pull/15786#issuecomment-2029520080 ## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) **Quality Gate passed** Issues  [8 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler=15786=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler=15786=false=true)  [86.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler=15786=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler=15786) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
