CalvinKirs opened a new pull request, #69177: URL: https://github.com/apache/doris/pull/69177
### What problem does this PR solve? Related PR: #69133 Backport #69133 to the legacy `branch-4.2` scan framework. Hive currently builds every split within a partition before publishing it. A consumer can also create an unbounded assignment queue, and a split fetch can exceed the backend's requested batch size. LIMIT queries therefore wait for, transmit, and retain ranges they never consume. - Enumerate Hive files and byte ranges lazily; bound producer batches with `split_batch_size=128` and queued batches per backend with `split_queue_capacity=32`. - Use one bounded queue factory regardless of which side creates a queue first. Return at most the BE-requested batch size and retain residuals through EOF. - Yield shared scheduling workers on full queues, resume after consumption, and retain partial backend-submission progress without replaying ranges. Serialize source advancement and cleanup. - Prefer eligible partition batching over the file-count gate. Preserve concurrent partition listing with bounded producer lanes and exclusive partition claims. Cancel EXPLAIN producers through statement cleanup. - Preserve the existing synchronous ACID/TABLESAMPLE paths, legacy six-argument constructor, public Iceberg resource hook, and nonthrowing stop contract. New producer/queue settings apply to Hive; other legacy generators retain their existing scheduling paths. 4.2 has no connector SPI or `VariableMgr.toMap` counterpart for the master-only parts of #69133. This backport uses `HiveScanNode`/`FileSplitter` directly and retains existing 4.2 table-resolution and explicit HMS-existence behavior. There is no BE protocol or storage-format change. #### Local performance Compare unmodified `branch-4.2` **e2a836ec154c94d069bca68be469802c7d8b62b0** with this backport **5904fd563ef65db8a77f0f4c9f1c7c006cb62ff9**. Each uses its own FE, the same **4.1.3-rc02 BE**, FE heap 2 GB, BE memory limit 4 GB, and the same local HMS/MinIO fixture: `split_hms.split_repro.files_10000` (10,000 nonpartitioned Parquet files, 1,000,000 rows, 18,840,000 bytes). Metadata and JVM are warm. SQL cache, BE file cache and query profiling are disabled. `enable_file_scanner_v2=false` is set on both FEs for compatibility with the 4.1.3 BE. BE split fetch size is 10; backport producer/queue settings use defaults 128/32. Measurements use one persistent client connection, sequential EXPLAIN and SELECT executions, and client execute plus full-fetch time. A 20 ms gap between statements is excluded from timing. Order: baseline A1, backport B1, backport B2, baseline A2. Every round warms each SQL 500 times and measures each SQL 100 times. The following medians and nearest-rank P95 values pool the two measured rounds (200 samples per SQL/version): | SQL | Unmodified 4.2 median | Backport median | Median reduction | Unmodified P95 | Backport P95 | |---|---:|---:|---:|---:|---:| | `EXPLAIN SELECT * ... LIMIT 10` | 33.728 ms | 3.689 ms | 89.1% | 48.630 ms | 8.413 ms | | `SELECT * ... LIMIT 10` | 297.578 ms | 8.758 ms | 97.1% | 387.745 ms | 22.324 ms | Per-round medians were 34.795/32.870 ms for baseline EXPLAIN and 3.920/3.465 ms for backport EXPLAIN; SELECT medians were 307.602/287.245 ms and 9.330/8.146 ms respectively. Both backport EXPLAIN medians meet the below-5-ms target. This is a shared-host FE + 4.1.3 BE comparison, rather than a benchmark of the 4.2 BE's V2 scanner. Invalid 2-GB BE warmup attempts were excluded. With `file_split_size=128`, the baseline plan contains **150,000 splits**; its LIMIT query fails the 4-GB BE memory limit while allocating the scan-range vector (reported process memory 5.66 GB). The backport returns 10 rows successfully, median **48.745 ms**, P95 **54.230 ms** over 20 samples after 10 warmups. No SELECT speedup ratio is claimed against that failed baseline execution. Separate EXPLAIN-only stress runs (10 warmups, 20 samples on each side) measured **489.828 ms → 5.450 ms**, with metadata/JVM prewarmed. The backport EXPLAIN-only cohort was collected after a restart and 500 additional EXPLAIN warmups; it is recorded separately from the interleaved stress SELECT cohort. ### Release note Hive can publish splits before completing a partition. Positive session settings `split_batch_size` and `split_queue_capacity` bound Hive producer batches and queued batches per backend. Split fetches honor the backend's requested size. ### Check List (For Author) - Test: **254 FE unit tests across 13 classes passed**, including Hive real routing/dispatch, million-file lazy prefix, both queue-creation orders, concurrent multi-backend identity checks, cancellation/rejection, and existing Hudi/Iceberg/MaxCompute tests. - Test: `build.sh --fe` and Checkstyle for every changed Java filename passed. Main compilation used the standard build; after a test-only literal fix, packaging and UT reused the compiled main classes with `skip.clean=true` and `maven.main.skip=true`. Production-source hashes match this committed tree. - Test: `test_hive_streaming_splits` generated its expected output through the standard regression runner and then passed verification; **zero skipped suites**. Its generated 236-line output is identical to the merged master output. - Test: Manual full COUNT/SUM, all 10,000 file identities, and 2,048-partition aggregates matched the baseline with producer batch 7, queue capacity 1, and BE fetch size 10. The million-file prefix UT uses a synthetic file list; a physical million-file benchmark is not claimed. - Compatibility: Public/protected API comparison of all seven modified existing production classes found zero removed signatures. - Behavior changed: Yes, incremental Hive split production, bounded queues and strict consumer batches. - Does this need documentation: No. (cherry picked from commit 6c8627ab97743194b03caf14c20faa68778d8b9b) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
