github-actions[bot] commented on code in PR #69040:
URL: https://github.com/apache/doris/pull/69040#discussion_r4356734240
##########
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java:
##########
@@ -1412,6 +1417,18 @@ public void setQueryId(TUniqueId queryId) {
}
}
+ /** Resolve each resource once for the current query without retaining it
in prepared expressions. */
+ public TS3Resource getOrRegisterS3Resource(String resourceName) {
+ return s3Resources.computeIfAbsent(resourceName,
Review Comment:
[P2] Refresh resource snapshots between routine-load tasks
This cache is cleared only by `setStartTime`/`reset`, but the shared-nothing
Routine Load scheduler retains one thread-local `ConnectContext` across tasks:
`RoutineLoadJob.plan` calls `setComputeGroup` and leaves `needCleanCtx=false`,
and subsequent `KafkaTaskInfo.rePlan` calls never start/reset that context. A
mapping such as `size = ELEMENT_AT(TO_FILE('r', uri), 'size')` therefore caches
`r` on its first task. After `ALTER RESOURCE` rotates the credentials,
`NereidsStreamLoadPlanner` keeps sending the old snapshot for later tasks;
revoking the old key makes those loads fail indefinitely on that scheduler.
Start a fresh resource snapshot at each load/task planning boundary while
retaining sharing within one task. The current refresh test explicitly calls
`setStartTime`, so it misses this caller.
##########
be/src/runtime/fragment_mgr.cpp:
##########
@@ -533,6 +533,9 @@ Status FragmentMgr::_get_or_create_query_ctx(const
TPipelineFragmentParams& para
if (params.__isset.ai_resources) {
query_ctx->set_ai_resources(params.ai_resources);
}
+ if (params.__isset.s3_resources) {
+ query_ctx->set_s3_resources(params.s3_resources);
Review Comment:
[P2] Handle S3 resources in external connector query plans
For an ordinary table with a STRING `uri` column, `/api/db/t/_query_plan`
admits this reduced plan:
```text
Project(id)
Filter(element_at(to_file('r', uri), 'size') > 0)
Scan(t: id, uri)
```
`ExpressionTranslator.visitToFile` registers `r` in the FE context, but
`TableQueryPlanAction.handleQuery` serializes a `TQueryPlanInfo` containing no
resource map. `exec_external_plan_fragment` then reconstructs pipeline
parameters without `s3_resources`, so this initialization is skipped and
`FunctionToFile::open` fails with `TO_FILE requires query resources`. The
output is scalar, so FILE Arrow-output restrictions do not prevent the trigger.
Handle resources for this separate execution entry point, or reject TO_FILE
while generating its opaque plan, and cover that round trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
