924060929 commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3838731145
##########
fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/sessions/FlightSqlConnectPoolMgr.java:
##########
@@ -74,7 +74,7 @@ public void unregisterConnection(ConnectContext ctx) {
// Finalize any Arrow Flight query whose coordinator was kept alive
across the
// GetFlightInfo -> DoGet phases (see #62259), releasing its resources
(e.g. external-table
// batch SplitSources and the query queue slot).
- ctx.closeFlightSqlDeferredExecutors();
+ ctx.sealAndCloseFlightSqlDeferredExecutors();
Review Comment:
这条评论涉及通用查询、任务、Streaming、Flight、Hive 或其他非本 PR 生命周期问题，不属于本 PR 仅处理 Hudi/Iceberg
资源关闭与泄露的范围。当前 head 已撤回对应旁支改动，本 PR 忽略该问题。
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertTask.java:
##########
@@ -173,24 +174,41 @@ protected void onFail(String errMsg) throws JobException {
@Override
public void cancel(boolean needWaitCancelComplete) {
super.cancel(needWaitCancelComplete);
- if (null != stmtExecutor) {
+ StmtExecutor executor = stmtExecutor;
+ if (null != executor) {
log.info("cancelling streaming insert task, job id is {}, task id
is {}",
getJobId(), getTaskId());
- stmtExecutor.cancel(new Status(TStatusCode.CANCELLED, "streaming
insert task cancelled"),
+ executor.cancel(new Status(TStatusCode.CANCELLED, "streaming
insert task cancelled"),
needWaitCancelComplete);
}
+ if (needWaitCancelComplete) {
+ awaitExecutionCompletion();
Review Comment:
这条评论涉及通用查询、任务、Streaming、Flight、Hive 或其他非本 PR 生命周期问题，不属于本 PR 仅处理 Hudi/Iceberg
资源关闭与泄露的范围。当前 head 已撤回对应旁支改动，本 PR 忽略该问题。
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/AbstractStreamingTask.java:
##########
@@ -94,34 +97,58 @@ public long getRunningBackendId() {
}
public void execute() throws JobException {
- while (retryCount <= MAX_RETRY) {
- try {
- before();
- run();
- onSuccess();
- return;
- } catch (Exception e) {
- if (TaskStatus.CANCELED.equals(status)) {
- return;
- }
- this.errMsg = e.getMessage();
- retryCount++;
- if (noRetry || retryCount > MAX_RETRY) {
- log.error("Task execution failed, job id {}, task id {},
noRetry {}, retry {}.",
- jobId, taskId, noRetry, retryCount, e);
- onFail(e.getMessage());
+ synchronized (executionCompletion) {
+ executionStarted = true;
+ }
+ try {
+ while (retryCount <= MAX_RETRY) {
+ try {
+ before();
+ run();
+ onSuccess();
return;
- }
- log.warn("execute streaming task error, job id is {}, task id
is {}, retrying {}/{}: {}",
- jobId, taskId, retryCount, MAX_RETRY, e.getMessage());
- } finally {
- // The cancel logic will call the closeOrReleased Resources
method by itself.
- // If it is also called here,
- // it may result in the inability to obtain relevant
information when canceling the task
- if (!TaskStatus.CANCELED.equals(status)) {
+ } catch (Exception e) {
+ if (TaskStatus.CANCELED.equals(status)) {
+ return;
+ }
+ this.errMsg = e.getMessage();
+ retryCount++;
+ if (noRetry || retryCount > MAX_RETRY) {
+ log.error("Task execution failed, job id {}, task id
{}, noRetry {}, retry {}.",
+ jobId, taskId, noRetry, retryCount, e);
+ onFail(e.getMessage());
+ return;
+ }
+ log.warn("execute streaming task error, job id is {}, task
id is {}, retrying {}/{}: {}",
+ jobId, taskId, retryCount, MAX_RETRY,
e.getMessage());
+ } finally {
+ // Only the scheduler worker that created this attempt's
ConnectContext may tear it down.
+ // A cancelling thread waits for this handoff instead of
racing before() and clearing fields
+ // while planning is still publishing them.
closeOrReleaseResources();
Review Comment:
这条评论涉及通用查询、任务、Streaming、Flight、Hive 或其他非本 PR 生命周期问题，不属于本 PR 仅处理 Hudi/Iceberg
资源关闭与泄露的范围。当前 head 已撤回对应旁支改动，本 PR 忽略该问题。
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/AbstractStreamingTask.java:
##########
@@ -94,35 +98,76 @@ public long getRunningBackendId() {
}
public void execute() throws JobException {
- while (retryCount <= MAX_RETRY) {
- try {
- before();
- run();
- onSuccess();
- return;
- } catch (Exception e) {
+ synchronized (executionCompletion) {
+ executionStarted = true;
+ executionOwner = Thread.currentThread();
+ }
+ try {
+ while (retryCount <= MAX_RETRY) {
+ Exception attemptFailure = null;
+ try {
+ before();
+ run();
+ } catch (Exception e) {
+ attemptFailure = e;
+ } finally {
+ // Only the scheduler worker that created this attempt's
ConnectContext may tear it down.
+ // A cancelling thread waits for this handoff instead of
racing before() and clearing fields
+ // while planning is still publishing them.
+ try {
+ closeOrReleaseResources();
+ } catch (RuntimeException cleanupFailure) {
+ if (attemptFailure == null) {
+ attemptFailure = cleanupFailure;
+ } else {
+ attemptFailure.addSuppressed(cleanupFailure);
+ }
+ }
+ }
+ if (attemptFailure == null) {
+ onSuccess();
Review Comment:
这条评论涉及通用查询、任务、Streaming、Flight、Hive 或其他非本 PR 生命周期问题，不属于本 PR 仅处理 Hudi/Iceberg
资源关闭与泄露的范围。当前 head 已撤回对应旁支改动，本 PR 忽略该问题。
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertTask.java:
##########
@@ -173,24 +174,57 @@ protected void onFail(String errMsg) throws JobException {
@Override
public void cancel(boolean needWaitCancelComplete) {
super.cancel(needWaitCancelComplete);
- if (null != stmtExecutor) {
+ StmtExecutor executor = stmtExecutor;
+ if (null != executor) {
log.info("cancelling streaming insert task, job id is {}, task id
is {}",
getJobId(), getTaskId());
- stmtExecutor.cancel(new Status(TStatusCode.CANCELLED, "streaming
insert task cancelled"),
+ executor.cancel(new Status(TStatusCode.CANCELLED, "streaming
insert task cancelled"),
needWaitCancelComplete);
}
+ if (needWaitCancelComplete) {
+ awaitExecutionCompletion();
Review Comment:
这条评论涉及通用查询、任务、Streaming、Flight、Hive 或其他非本 PR 生命周期问题，不属于本 PR 仅处理 Hudi/Iceberg
资源关闭与泄露的范围。当前 head 已撤回对应旁支改动，本 PR 忽略该问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
