Doris-Breakwater commented on issue #69926: URL: https://github.com/apache/doris/issues/69926#issuecomment-6560157651
The attachment points to a **likely query-correctness defect in the V2 JSON reader when no physical file columns are projected**. The evidence is strong enough to focus investigation on row-count preservation in that path, but the exact runtime projection and a reproducer/A-B result are still needed to close the diagnosis. Suggested triage: BE external scan / Hive JSON / wrong results. What is established by `1111.xlsx`: - Both queries report `doris-4.1.5-release-917b20be838`, `Is Cached: No`, and `UseScannerV2: true`. - Both plans select `partition=1/9`, with 15 splits and `totalFileSize=489359939`. Both explicitly show **`pushdown agg=NONE`**. This capture therefore does not support blaming the external COUNT pushdown optimization. - The count profile reads **522.64 MB**, opens 15 files, and reports `ScanRows=0`, `RowsProduced=0`, and `FinalizeBlockTime=0ns`. `EmptyFileNum`, `NotFoundFileNum`, and `RuntimeFilterPartitionPrunedRangeNum` are all zero. Bytes are being read, but no rows reach the aggregate. - The `SELECT * ... LIMIT 10` profile reports 4,064 scan rows and returns 10 rows through the same V2 JSON reader. It proves that matching rows exist; a limited query does not establish the full partition count or independently verify the approximately 1.148 million rows mentioned in the screenshot. I checked the source at the exact reported commit, `917b20be83878e98f94e3ebf7efccc80346e3e50`. The following mechanism matches the observed difference: 1. The count plan projects a constant above the partition filter. Unlike `SELECT *`, it need not materialize a business column from the JSON file. [`FileScannerV2::_build_projected_columns`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/exec/scan/file_scanner_v2.cpp#L1077-L1128) separates partition columns from `_file_slot_descs`. 2. The column mapper supplies partition values as constants and adds physical read columns only for mappings with a file-local column ID. Thus a partition-only projection can produce an empty file block. See [`column_mapper.cpp`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/format_v2/column_mapper.cpp#L2279-L2283). 3. [`JsonReader::get_block`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/format_v2/json/json_reader.cpp#L300-L364) derives its returned row count from `file_block->rows()`. With no requested file columns, parsing an object does not append any column values; an empty block has zero rows. The reader also records `ScanRows` **before** applying its filters, so the observed zero is consistent with losing the row count during materialization, rather than normal rows subsequently being rejected by the partition predicate. 4. [`TableReader::get_block`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/format_v2/table_reader.h#L337-L365) skips `finalize_chunk()` whenever `current_rows == 0`. Consequently the partition constants never receive rows, matching `FinalizeBlockTime=0ns` and the final zero count. This is a code-supported diagnosis, not a claim that I reproduced the query against your data. In particular, the plan's `operativeCols` field alone is not a dump of the physical reader's requested columns. The most useful next checks are: 1. In one session, keep the partition files unchanged and compare `SELECT etl_date ... LIMIT 10` with `SELECT id, etl_date ... LIMIT 10`, using the same partition predicate and a real file column such as `id`. Keep V2 enabled and external COUNT pushdown disabled. If only the second query returns rows, that directly supports the empty-file-projection diagnosis. Please attach both plans and plain-text profiles. 2. Test the original count with these **session-only** settings, recording the previous values first: ```sql SET enable_profile = true; SET enable_count_push_down_for_external_table = false; SET enable_file_scanner_v2 = false; -- Run the original COUNT(*) and EXPLAIN for the same partition. ``` Confirm `UseScannerV2: false` in the new profile and `pushdown agg=NONE` in the plan, and compare the result with Hive over the same files. Restore the previous settings afterward. This is an **unverified workaround candidate**: the [V1 JSON reader](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/format/json/new_json_reader.cpp#L205-L230) maintains `read_rows` independently, and the [V1 scanner](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/exec/scan/file_scanner.cpp#L587-L603) uses it to fill partition columns on the non-COUNT-pushdown path. Disabling COUNT pushdown alone does not address the V2 mechanism above. 3. Supply a sanitized full Hive DDL (including SerDe properties), Doris `DESC`, a few representative JSON records, and Hive's exact count. If the comparison is inconclusive, capture the scan's required slots/file-column request and query-ID-correlated BE logs; also confirm all participating FE/BE builds match the reported version. For maintainers, the fix direction is to preserve the number of valid JSON records even with zero physical output columns, or provide a correct fallback. Regression coverage should include partition-only versus file-column projections, COUNT and scanner-version combinations, empty input/objects, split boundaries, and existing malformed-JSON semantics. `COUNT(id)` is not a generally equivalent replacement for `COUNT(*)` because it excludes NULLs. No permanent fix or fixed release has been verified in this investigation. Breakwater-GitHub-Analysis-Slot: slot_0f7168a6ada3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
