hanke580 opened a new issue, #68116: URL: https://github.com/apache/doris/issues/68116
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version **Affected:** Apache Doris **4.1.4** — BE reports `doris-4.1.4-rc04-ad35a140c7f` (the newest released build; images `apache/doris:fe-4.1.4` + `be-4.1.4`, pushed 2026-09-11). Single FE + single BE, Nereids planner, otherwise default configuration, Linux x86_64. **Also affected: current `master`.** There is no nightly/master Doris image, so I built `apache/doris` master commit **`c4dee4bd5e82c6f0f380e0243cbaf265999e6a6a`** from source with `apache/doris:build-env-ldb-toolchain-latest` and ran the identical battery against a single-FE/single-BE cluster from that build. The clone's HEAD was verified equal to GitHub's `master` HEAD (2026-09-17T03:34:48Z). BE reports `doris-0.0.0-Unknown` because a local build carries no version stamp; the shipped default `batch_size` is 8160 there too. **master reproduces every measurement identically to 4.1.4** — not approximately, the same counts row for row: | measurement | 4.1.4 | master `c4dee4bd` | |---|---:|---:| | `first_value(varchar)` wrong @ `batch_size` 4 / 2 / 1 | 4 / 10 / 9 | **4 / 10 / 9** | | longest bogus string @ 4 / 2 / 1 | 229 / 525 / 772 | **229 / 525 / 772** | | `first_value(int)` wrong @ `batch_size` 38 / 37 / 4 / 1 | 392 / 908 / 15 888 / 17 180 | **392 / 908 / 15 888 / 17 180** | | onset rule at partition size 10 000 / 5 000 / 2 000 | `bs` 38 / 19 / 7 | **38 / 19 / 7** | | 2 560 000-row partition @ default `batch_size` 8160 | 454 720 wrong | **454 720 wrong** | | 1 280 000-row partition @ default `batch_size` 8160 | 0 | **0** | | `nth_value` string length | `total_length=4294967303` | **`total_length=4294967296`** (2³², message now also prints `limit=4294967295`) | and **master crashed the BE** during the run. Because it is a locally built binary the trace carries full source lines, which is the clearest statement of the bug available: ``` *** SIGSEGV invalid permissions for mapped object (@0x7f15c87fd000) received by PID 2333 *** 2# memcpy at ./be/build_Release/../src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219 3# doris::ColumnStr<unsigned int>::insert_from(doris::IColumn const&, unsigned long) at ../src/core/column/column_string.h:188 4# doris::IAggregateFunction::insert_result_into_range(char const*, doris::IColumn&, unsigned long, unsigned long) const at ../src/exprs/aggregate/aggregate_function.h 5# doris::AnalyticSinkLocalState::_insert_result_info(long, long) at ./be/src/exec/operator/analytic_sink_operator.cpp:404 6# doris::AnalyticSinkLocalState::_get_next_for_unbounded_rows(long, long) at ./be/src/exec/operator/analytic_sink_operator.cpp:248 8# doris::AnalyticSinkOperatorX::sink_impl(doris::RuntimeState*, doris::Block*, bool) at ./be/src/exec/operator/analytic_sink_operator.cpp:759 ``` A `memcpy` inside `ColumnStr::insert_from`, reached from the analytic sink writing out an aggregate's saved value — i.e. copying a string through an offset that no longer describes the row it was saved for. No fix is in flight: the most recent commit touching `analytic_sink_operator.cpp` is #67274 (2026-08-31) and the most recent touching `aggregate_function_window.h` is #64864 (2026-06-26); neither addresses the saved-pointer invalidation. ### What's Wrong? `first_value()` over a window silently returns values that are not in the window — and on a `VARCHAR` column, values that are not in the table at all. On the same data the BE can also abort the query with a bogus 4 GB string length, or crash outright. The trigger is that the analytic operator evicts input rows it believes are no longer needed, while `first_value` is still holding a **raw pointer plus an absolute row offset** into exactly the columns being evicted. #### 1. `first_value(varchar)` returns a string that is nowhere in the table With `PARTITION BY c`, `first_value(c)` must equal the row's own `c` — every partition contains one distinct value, so this is an exact oracle needing no reference engine. 20 000 rows, every value at most 8 characters: | `batch_size` | rows wrong | longest bogus value returned | |---:|---:|---:| | 65535 | 0 | — | | 8160 (shipped default) | 0 | — | | 8 | 0 | — | | **4** | **4** | **229 chars** | | **2** | **10** | **525 chars** | | **1** | **9** | **772 chars** | A sample row: `c = 'xyz'`, so the correct answer is `'xyz'`, and Doris returns ``` azczazaczcbczcbczcbczyaxabxxaaxbbxabaxabcayabxacazayxaxcxyzxaxyxxxx… (229 characters) ``` That is a run of adjacent values read out of the string column through a wrong offset/length pair — the correct 3-character value is visible inside it. #### 2. `first_value(int)` is wrong for most of the table Same shape on an INT column, ground truth computed in Python (`PARTITION BY e` with `e` boolean, so two partitions of ~10 000 rows): | `batch_size` | rows wrong (of 20 000) | |---:|---:| | 65535 / 8160 / 64 / 40 | 0 | | 39 | 0 | | **38** | **392** | | **37** | **908** | | **4** | **15 888** (79%) | | **1** | **17 180** (86%) | No warning, no error — the query simply returns wrong numbers for most rows. #### 3. It reproduces at the shipped `batch_size` on an ordinary table `batch_size` only makes it cheap to demonstrate. What actually matters is the **partition size in blocks**, so the same defect is reached with `batch_size` left completely alone (shipped default **8160**) by making one partition large — the threshold being `257 * 8160 = 2 097 120` rows. Single partition, nothing configured: | rows in the partition | wrong at default `batch_size` (8160) | wrong at `batch_size = 65535` | |---:|---:|---:| | 160 000 – 1 280 000 | 0 | 0 | | **2 560 000** | **454 720** (18%) | 0 | The boundary lands exactly where the `257 * batch_size` rule predicts (1 280 000 < 2 097 120 < 2 560 000), and raising `batch_size` to 65535 moves the threshold out of reach again, which isolates the block count as the cause. One caveat worth knowing when reproducing: the block count is produced by the **scan**, so it depends on how the table is physically laid out, not only on `batch_size`. On a table I had built by repeated `INSERT INTO … SELECT` doubling (many small rowsets) the same 2 560 000-row partition was wrong at `batch_size = 65535` as well, where a table of the same size written by a single `INSERT … SELECT` was clean at 65535. So a fragmented table reaches 257 blocks with fewer rows, and raising `batch_size` does not reliably protect it. #### 4. The exact trigger The onset is not fuzzy. Corruption begins exactly when ``` partition_size > 257 * batch_size ``` which is `block_num = 256` in `_remove_unused_rows()`, plus the `+ 1` in its guard. Verified by fixing the partition size and walking `batch_size` across the predicted boundary: | partition size | predicted onset | `bs` = pred+1 | `bs` = pred | `bs` = pred−1 | |---:|---:|---:|---:|---:| | 10 000 | `bs` ≤ 38 | 0 wrong | **392 wrong** | 908 wrong | | 5 000 | `bs` ≤ 19 | 0 wrong | **392 wrong** | 1 424 wrong | | 2 000 | `bs` ≤ 7 | 0 wrong | **1 940 wrong** | 4 520 wrong | Three for three, on the row exactly. #### 5. The same corruption as an error, and as a crash `nth_value(varchar, 3)` over `ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING`, 20 000 rows, `batch_size = 4`: ``` errCode = 2, detailMessage = [E-3113] string column length is too large: total_length=4294967303, element_number=4, rows=0 ``` `4294967303` is 2³² + 7 — the string length arithmetic wrapped a 32-bit counter, which is the same wrong offset/length pair that produces the 229-character `first_value` answers. The same shape **took the BE down** on both builds — on 4.1.4 at `batch_size = 1`, and on the master build during the battery. Stack from the 4.1.4 BE log (the master one, with full source lines, is in the `Version` section): ``` 2# doris::ColumnStr<unsigned int>::insert_from(doris::IColumn const&, unsigned long) at be/src/core/column/column_string.h:71 3# doris::IAggregateFunction::insert_result_into_range(char const*, doris::IColumn&, unsigned long, unsigned long) const at be/src/exprs/aggregate/aggregate_function.h:252 4# doris::AnalyticSinkLocalState::_insert_result_info(long, long) at be/src/exec/operator/analytic_sink_operator.cpp:406 5# doris::AnalyticSinkLocalState::_get_next_for_unbounded_rows(long, long) at be/src/exec/operator/analytic_sink_operator.cpp:250 6# doris::AnalyticSinkLocalState::_execute_impl(doris::RuntimeState*) 7# doris::AnalyticSinkOperatorX::sink_impl(doris::RuntimeState*, doris::Block*, bool) ``` Frame 2 is the aggregate's saved value being copied out: `to.insert_from(*_ptr, _offset)`. It is reading through a pointer/offset that no longer describes the row it was saved for. ### What You Expected? `first_value(c) OVER (PARTITION BY c …)` must return the row's own `c` for every row, and `first_value(a)` must return the first `a` of the partition in window order — regardless of `batch_size`, partition size, or whether the operator decided to release memory mid-partition. The answer must not depend on how the input was split into vectorised blocks. At minimum, a query must not silently return values that do not exist in the input, and must not crash the backend. ### How to Reproduce? Single FE + single BE, default configuration: ```bash docker network create --subnet=172.30.0.0/24 doris-net docker run -d --name fe --network doris-net --ip 172.30.0.2 \ -e FE_SERVERS="fe1:172.30.0.2:9010" -e FE_ID=1 -p 9030:9030 apache/doris:fe-4.1.4 docker run -d --name be --network doris-net --ip 172.30.0.3 \ -e FE_SERVERS="fe1:172.30.0.2:9010" -e BE_ADDR="172.30.0.3:9050" apache/doris:be-4.1.4 ``` **A. Smallest form — exact oracle, no reference engine needed.** Load 20 000 rows of `(id INT, c VARCHAR(255))` where `c` is a random string of at most 8 characters, then: ```sql SET batch_size = 4; -- 20 000 rows / 4 = 5 000 blocks per scan SELECT id, c, first_value(c) OVER (PARTITION BY c ORDER BY c, id ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS w FROM t HAVING w <> c; -- must be empty; returns rows whose w is 229 chars long ``` Because the window partitions by `c`, `w` can only be `c`. Any returned row is a wrong answer. `SET batch_size = 65535` on the same table returns nothing, confirming the data is fine. **B. At the shipped `batch_size`, no session variable touched.** Build `(id INT, a INT, g BIGINT, e INT)` with `e = 0` on every row, so the whole table is one partition, and grow it past `257 * 8160 = 2 097 120` rows: ```sql -- batch_size left at its shipped default of 8160 SELECT count(*) FROM ( SELECT id, first_value(a) OVER (PARTITION BY e ORDER BY g, id ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS w FROM big ) s WHERE w <> (SELECT a FROM big ORDER BY g, id LIMIT 1); -- 1 280 000 rows -> 0 (below 257*8160) -- 2 560 000 rows -> 454 720 (must be 0) ``` `SET batch_size = 65535` returns 0 for both, which isolates the block count as the cause. **C. The crash / 4 GB length.** On the 20 000-row varchar table: ```sql SET batch_size = 4; SELECT id, nth_value(c, 3) OVER (PARTITION BY b % 13 ORDER BY c, id ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS w FROM t; -- [E-3113] string column length is too large: total_length=4294967303 SET batch_size = 1; -- took the BE down in my run ``` A reproduction script with the ground-truth oracles for all of the above is attached to this report. ### Anything Else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
