sollhui opened a new pull request, #66099:
URL: https://github.com/apache/doris/pull/66099
### What problem does this PR solve?
Issue Number: None
Problem Summary: Routine Load treated a task that only reached its time
limit as evidence of backlog. Continuous low-throughput input could therefore
keep the task on the enlarged adaptive interval and delay Kafka offset commits.
This change:
- starts new Kafka and Kinesis Routine Load tasks with the job-configured
batch settings;
- enters adaptive batching only after the current task reaches its row or
byte limit;
- keeps adaptive batching only when an adaptive task reaches its enlarged
row or byte limit;
- restores the user-configured interval when an adaptive task times out with
low data volume.
### Release note
Routine Load no longer extends the batch interval for continuous
low-throughput input that does not reach the row or byte threshold.
### Check List (For Author)
- Test: Unit Test
- `./run-fe-ut.sh --run
org.apache.doris.load.routineload.KafkaRoutineLoadJobTest` (10 tests passed)
- `./run-fe-ut.sh --run
org.apache.doris.load.routineload.KinesisRoutineLoadJobTest` (9 tests passed)
- Behavior changed: Yes. Low-throughput tasks keep the user-configured batch
interval; high-throughput tasks still use adaptive batching.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
