morningman opened a new pull request, #70827:
URL: https://github.com/apache/doris/pull/70827

   ### Background
   
   On master, `lance` catalogs are served by the `fe-connector-lance` plugin: 
FE lists the fragments (data files) of the pinned dataset version and plans one 
scan range per fragment, and BE reads them through lance-c. A dataset may carry 
scalar indexes (BTREE, BITMAP, LABEL_LIST) split into segments, each covering 
some fragments. lance-c can apply a Substrait filter, a row limit and one index 
segment per scan, and master's BE reader (#69581) passes them on from 
`TLanceScanParams` and `TLanceFileDesc`, but the plugin sends none of them yet.
   
   This is the third of four PRs porting the Lance read path (#70680). It is 
stacked on the second, #70725, which is still in review: until that merges, 
GitHub also shows its commits here; review only the last commit.
   
   ### What problem does this PR solve?
   
   Issue Number: #70680
   Related PR: #70725 (REST catalogs and OSS, the second PR; this builds on 
it), #70681 (the plugin, the first PR), #70369 (ARRAY literals as element 
lists, which `arrays_overlap` needs), #70372 (marks UDF calls, which this keeps 
in Doris), #69581 (the BE Lance reader, which applies what this PR sends), 
#65730, #66608, #66999, #67051, #67650, #68687 (the branch-4.x PRs this ports)
   PR Type: Feature
   
   Problem Summary:
   
   On master, a Lance catalog scan leaves Lance all the work it could skip: 
filters run in Doris after lance-c has read every row, lance-c never sees the 
LIMIT, and COUNT(*) reads every fragment, while 4.1 and 4.2 push all three into 
Lance. This PR ports 4.2's pushdown to the plugin. Conjuncts that Lance 
evaluates exactly as Doris does go to lance-c as one Substrait filter and leave 
the Doris plan; LIMIT reaches every range when no conjunct stays in Doris; 
COUNT(*) without a filter comes from the manifest's row counts; and a pushed 
filter that one scalar index can drive gets one range per index segment. For 
example, on the regression table `doris.predicate_pushdown` (11 rows in 2 
fragments), measured on this branch:
   
   | Query | EXPLAIN of the Lance scan |
   | --- | --- |
   | `... WHERE int32_value = 10 LIMIT 1` | 
`lancePushdownPredicate=(int32_value = 10)`, `lanceLimit=1`, no `PREDICATES:` 
line |
   | `... WHERE CAST(int32_value AS STRING) = '10' LIMIT 1` | `PREDICATES: 
(CAST(int32_value[#4] AS text) = '10')`, no `lanceLimit`: Doris still filters, 
as it filtered every conjunct before this PR |
   | `SELECT count(*) ...` | `pushdown agg=COUNT (11)` |
    Most of the diff is tests and their results (68% of the added lines). Query 
results do not change.
   
   Original authors: @zhangstar333, @Jay-ju and @Gabriel39.
   
   ### Details
   
   - **Change map:** +8036 −66 in 115 files: production code +2568 −45, tests 
+3290 −21 (FE unit tests +1786, regression suites +1177, fixture builders +320, 
and 73 dataset files of 144 KiB, mostly binary), generated `.out` +2159, build 
+19. The production code, all in the plugin except one session variable, in 
reading order:
     1. Predicate translation (+1,135): `LancePredicateConverter` (4.2's rules, 
rewritten for `ConnectorExpression`), `LanceSubstraitSerializer` and 
`lance_functions.yaml` (4.2's).
     2. Index metadata (+493 −5): `LanceDatasetIndexDiscovery`, 
`LanceIndexSegmentInfo` and `LanceIndexSegmentGroup` (4.2's, without the vector 
and full-text parts), and `LanceIndexMetadata`, read once per statement through 
`LanceCatalogClient` and `LanceStatementScope`.
     3. Planning (+727): `LanceScalarIndexPlanner` (which index segment can 
drive the filter), `LanceScanPlanner` and `LanceSplitBuilder` (ranges, COUNT 
carriers, debug grouping), `LanceScanPlan` (what EXPLAIN prints).
     4. Ranges and the provider (+197 −40): `LanceScanRange` (index segment, 
`use_scalar_index`, limit, COUNT carrier) and `LanceScanPlanProvider` (which 
conjuncts were pushed, the filter in `TLanceScanParams`, EXPLAIN).
     5. fe-core (+16): the session variable `lance_fragments_per_split`, 
declared there as the Fluss connector's `fluss_union_read_mode` is. Build: 
`io.substrait:core` 0.40.0 (Apache-2.0, the version 4.x pins).
   - **Scope:** Pushed, as on 4.2: comparisons (`<=>` included) between a 
column and a literal, `[NOT] IN` lists without NULL, `IS [NOT] NULL`, `LIKE` 
without a backslash, `starts_with`, `ends_with`, boolean columns, `AND` / `OR` 
/ `NOT` whose operands all convert, and `array_contains` / `arrays_overlap` on 
string arrays with non-NULL needles; on bool, integer, float32/64, decimal128, 
utf8, date32 and zone-less timestamp(s/ms/us) columns. Only builtin calls are 
translated: a Java or Python UDF with one of these names stays in Doris, as on 
4.2. The rest stays in Doris under `PREDICATES:`, such as CASTs, comparisons 
with NULL and other column types. Index segments: BTREE, BITMAP and LABEL_LIST, 
for a positive predicate within lance-c's bounds (128 nodes, depth 32).
   - **Design:** A conjunct is translated whole or not at all, and only exact 
translations are reported pushed 
(`ScanNodePropertiesResult.withPushdownTracking`, as the Elasticsearch 
connector does), so the engine drops exactly those from the Doris residuals, as 
4.2 did. Where the SPI forces a different shape from 4.2:
     - CASTs stay in Doris (`supportsCastPredicatePushdown` returns false), as 
on 4.2: the SPI default strips the CAST and would let Lance compare in the 
column's own type.
     - UDFs: 4.2 checked `TFunctionBinaryType.BUILTIN`; here the converter 
refuses a `ConnectorFunctionCall` whose `isBuiltin()` is false (#70372).
     - No plan state on the provider: the engine memoizes it by table handle, 
which the snapshot pin replaces, so EXPLAIN (`getScanNodePropertiesResult`) and 
the ranges (`planScan`) are planned deterministically from the same inputs.
     - COUNT(*) carriers: 4.2 makes min(fragments, instances × backends), and 
the SPI gives neither count, so 10,000 rows or more get one carrier per 
fragment and fewer get one. Each carrier keeps its fragments for a BE that 
falls back to scanning.
     - Index metadata is read only when a segment could be planned (4.2 reads 
it for every scan), so `DESC` and plain scans don't fail on an unreadable index.
   - **Usage:** Nothing to enable:
   
     ```sql
     EXPLAIN SELECT row_id FROM lance_s3.db.items WHERE row_id >= 2 AND label = 
'a' LIMIT 10;
     --   lanceFragmentGrouping=FRAGMENT
     --   lanceLimit=10
     --   lancePushdownPredicate=(row_id >= 2) AND (label = 'a')
     -- and no PREDICATES: line, since nothing is left for Doris. When one 
scalar index drives the
     -- filter, lanceFragmentGrouping=INDEX_SEGMENT and 
lanceGroupingIndex=<index> appear instead.
   
     EXPLAIN SELECT COUNT(*) FROM lance_s3.db.items;  -- pushdown agg=COUNT 
(<rows>)
   
     SET lance_fragments_per_split = 2;  -- debugging: two fragments per range, 
no index segments
     ```
   
   - **Default behavior:** On for every Lance catalog scan; 
`lance_fragments_per_split` defaults to 0 (automatic grouping) and rejects 
negative values.
   - **Compatibility:** No Thrift, Protobuf or persisted-metadata change, and 
no connector SPI change, so the plugin API version stays as it is: the plugin 
fills fields that master's BE has read since #69581 and 4.2's FE sends 
(`lance_substrait_filter`; `limit`, `index_segment_uuids` and 
`use_scalar_index` in `TLanceFileDesc`; the generic `table_level_row_count`). 
`lance_fragments_per_split` keeps 4.2's name, default and checker, but is 
visible: the engine hands connectors only visible session variables 
(`VariableMgr.toMap`).
   - **Follow-up plan:** PR 4 adds the session and table-access caches and the 
profile counters; vector and full-text search, index inspection and the file 
TVF follow, in the same issue. Later 4.2 changes to these files (#68813, #69625 
and the search work) are ported with the next batch.
   
   ### Testing
   
   - `fe-connector-lance` unit tests, 213, locally on this commit with 
lance-core's JNI library loaded (none skipped): 4.2's converter cases on 
`ConnectorExpression` input, its two Java-UDF cases included, plus new ones for 
DECIMAL exactness, integer ranges and UDFs named like each translated builtin 
(34); all 9 of 4.2's scalar-index planner cases; 13 of its scan-node cases (the 
non-search planning ones; one now expects a COUNT carrier per fragment); its 
two Thrift contract cases; and the provider's residual, LIMIT and COUNT rules.
   - 9 mutations, each run against the unchanged tests: every one fails its 
target tests (a UDF translated by name, a truncated DECIMAL, NULL needles 
dropped, `<=>` without `IS NOT NULL`, an escaped LIKE pushed, no integer range 
check, a LIMIT past a residual, every conjunct claimed pushed, COUNT despite a 
filter).
   - fe-core, 55 tests: `SessionVariablesTest` (the variable reaches connectors 
and rejects negative values), `ExprToConnectorExpressionConverterTest`, 
`ExprToConnectorFunctionCallTest` and `PluginDrivenScanNodeCastPredicateTest`. 
The FE build ran checkstyle and the connector import gate.
   - Regression, locally, with FE and an ASAN BE built from this commit, the 
committed REST stub, and a host S3 endpoint that verifies SigV4 and serves the 
committed fixture: the six new suites, PR 1's `test_lance_catalog_all_types` 
and PR 2's `test_lance_rest_catalog` pass against the committed `.out` files (8 
suites, 0 failed, 0 skipped). The new `.out` files are branch-4.2's results 
block for block: int 176/176, scalar 178/178, optimize_count 10/10, 
runtime_filter 1/1 (its two vector-search blocks come with search). In CI, the 
External pipeline runs them.
   
   ### Notes for Reviewers
   
   - Once #70725 merges, this branch is rebased onto master with only its last 
commit.
   - Master's plugin scan node prints Doris residuals as `PREDICATES:`, not 
4.x's `predicates:`, so the 12 ported checks use that spelling (`notContains 
"predicates:"` would always pass); `test_lance_runtime_filter_pushdown` keeps 
only 4.2's non-search case and asserts `CONNECTOR: lance`.
   - DECIMAL: 4.2 relies on `ExpressionCreator.decimal` throwing, but it 
truncates to the column's scale (`12.505` becomes `12.50`), so a literal is 
pushed only when the column's type holds it exactly; the unit test fails 
without the check.
   - `arrays_overlap` also requires a string element type: LARGEINT, IPv4 and 
JSON elements arrive as Strings too (#70369).
   - The fixture is committed instead of built by Spark: the `doris` 
namespace's `__manifest`, `predicate_pushdown` (4.2's rows, two appends), and 
4.2's `vs_ivf_pq_f32` (read by `test_lance_fragment_grouping`), 
`multi_frag.lance` and `predicate_arrays/` unchanged. 
`lance_build_predicate_pushdown.py` rebuilds or `--check`s it with 4.2's pinned 
writer.
   - The extension URI of `array_has` names the plugin's 
`lance_functions.yaml`; lance-c resolves functions by name.
   
   ### Release note
   
   The session variable `lance_fragments_per_split` (default 0) is a debugging 
override for Lance catalog scans: a positive value groups that many fragments 
into each scan range and turns index-segment ranges off. Unlike 4.1 and 4.2, 
`SHOW VARIABLES` lists it.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No
   - Pre-submission status: Partially verified: the regression suites ran 
locally against a host S3 substitute and the REST stub, not yet in the External 
pipeline
   
   ### AI Disclosure
   
   - Tool: Claude Code (Claude Opus 5.5)
   - Generated: code, tests, the fixture builder, this description
   - Human review: The author reviewed part of the changes.
   - Prompts: The author asked to port the branch-4.2 Lance FE read path to 
master as a connector plugin in four PRs, of which this is the third; approved 
handing ARRAY literals to connectors as element lists (#70369) and marking UDF 
calls (#70372) in separate PRs first; had the regression fixture committed as a 
dataset rather than built by Spark; and asked to build this PR on #70725 
without waiting for it to merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to