Gabriel39 opened a new pull request, #70154:
URL: https://github.com/apache/doris/pull/70154
### What problem does this PR solve?
Issue Number: close #69926
Problem Summary:
Hive JSON queries that only need partition columns can return zero rows even
when the selected files contain records. For example, `SELECT * ... WHERE dt =
'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero.
The V2 JSON reader derived its row count from the output block, which has no
columns after physical field pruning.
Track accepted JSON records independently of materialized columns, including
batch progress, array expansion, error rollback, and ignore-malformed NULL
rows. Preserve JSON-path record selection for empty projections, and let the
existing TableReader materialize partition values using the returned row count.
Add ten BE tests covering empty file schemas and projections, arrays, JSON
roots/paths, malformed input, batch boundaries, EOF, and actual TableReader
partition materialization. Add a Hive SQL regression using the existing
partitioned JSON fixture for COUNT, partition projections, grouping, and
file-column filters with external COUNT pushdown enabled and disabled.
### Release note
Fix incorrect zero counts and missing rows when querying only partition
columns of Hive JSON tables.
### Check List (For Author)
- Test
- [x] Unit Test: 151 ASAN tests passed, including all JSON reader tests,
TableReader tests, and CSV/Text empty-projection controls. Before the fix, nine
of the ten new tests failed with missing rows; the empty-input control passed.
- [x] Regression test: added `test_hive_json_partition_only`; Groovy
compilation passed. Full execution against the external Hive fixture is pending
CI.
- Behavior changed:
- [x] Yes. Empty JSON file projections preserve the number of accepted
records and produce the correct partition rows and aggregates.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
