0AyanamiRei opened a new pull request, #65216:
URL: https://github.com/apache/doris/pull/65216
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
This PR exposes historical Stream Load records through the existing
`information_schema.load_jobs` table.
Before this change, Doris could show Stream Load records through `SHOW
STREAM LOAD` when BE Stream Load record collection was enabled, but
`information_schema.load_jobs` only returned ordinary load jobs. This made
`load_jobs` less useful as a unified load-observability surface.
This PR keeps Doris's existing Stream Load record workflow:
- BE stores completed Stream Load records when `enable_stream_load_record`
is enabled.
- FE `StreamLoadRecordMgr` periodically pulls those records from BE.
- `SHOW STREAM LOAD` continues reading the same FE cache.
The change appends mapped Stream Load records to
`FrontendServiceImpl.fetchLoadJob()` so users can query them from
`information_schema.load_jobs`. Stream Load-specific details are placed into
existing JSON columns:
- `TASK_INFO`: `Db`, `Table`, `ClientIp`
- `JOB_DETAILS`: `TotalRows`, `LoadedRows`, `FilteredRows`,
`UnselectedRows`, `LoadBytes`, `BeginTxnTimeMs`, `StreamLoadPutTimeMs`
The existing `information_schema.load_jobs` schema is unchanged, no
`TFetchLoadJobRequest` pushdown fields are added, and no new SR-style FE Stream
Load task lifecycle is introduced.
Stream Load rows in `information_schema.load_jobs` require the same BE
record collection and FE polling dependency as `SHOW STREAM LOAD`.
### Release note
Support querying Stream Load records from `information_schema.load_jobs`
when Stream Load record collection is enabled.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Test details:
```bash
/data/data3/huangruixin/include/src-master/apache-maven-3.9.9/bin/mvn \
-pl fe-core -am \
-Dtest=org.apache.doris.load.StreamLoadRecordMgrTest \
-DfailIfNoTests=false test -Dskip.doc=true
```
```bash
DORIS_HOME=/data/data3/huangruixin/main/doris-master/doris \
/data/data3/huangruixin/include/src-master/ldb_toolchain/bin/ninja \
-C be/build_ASAN \
src/load/CMakeFiles/Load.dir/stream_load/stream_load_context.cpp.o
```
```bash
./build.sh --fe --be -j48
```
```bash
./run-regression-test.sh --run -d load_p0/stream_load -s
test_stream_load_record \
-c
'jdbc:mysql://127.0.0.1:61930/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round'
\
-ha 127.0.0.1:61030
```
```bash
./run-regression-test.sh --run -d job_p0/job_system_table -s
test_load_job_info_system_table \
-c
'jdbc:mysql://127.0.0.1:61930/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round'
\
-ha 127.0.0.1:61030
```
- Behavior changed:
- [ ] No.
- [x] Yes. Stream Load records can now appear in
`information_schema.load_jobs` as `TYPE = 'STREAM_LOAD'` rows after FE pulls
them from BE.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
