vaquarkhan commented on PR #57016:
URL: https://github.com/apache/spark/pull/57016#issuecomment-4901074811
I can see two issues in code
**1 BUG - ConnectResource.scala, prepareExecutionData**
jobIds = info.jobId.toSeq.sorted and sqlExecIds =
info.sqlExecId.toSeq.sorted sort numeric ids as strings, so e.g. ["2","10"] is
returned as ["10","2"]. The unit test passes only because the chosen values
("1","2" and "0","10") sort identically lexicographically and numerically, so
it hides the defect. Use .sortBy(_.toInt) for both (or document the order is
intentionally lexicographic).
**2 CRITICAL - ConnectResource.scala, paginate**
Pagination slices after getSessionList/getExecutionList, which call
KVUtils.viewToSeq(store.view(...)) and materialize the entire store on every
request. So ?offset=0&length=10 still loads all N sessions/operations, and a
bare GET /connect/operations (default length=-1) returns an unbounded payload.
On a long-lived Connect server this is an OOM/latency risk. Please push
offset/length into the KVStoreView (skip/max) and cap the default length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
