morningman opened a new pull request, #70827:
URL: https://github.com/apache/doris/pull/70827
### Background
On master, `lance` catalogs are served by the `fe-connector-lance` plugin:
FE lists the fragments (data files) of the pinned dataset version and plans one
scan range per fragment, and BE reads them through lance-c. A dataset may carry
scalar indexes (BTREE, BITMAP, LABEL_LIST) split into segments, each covering
some fragments. lance-c can apply a Substrait filter, a row limit and one index
segment per scan, and master's BE reader (#69581) passes them on from
`TLanceScanParams` and `TLanceFileDesc`, but the plugin sends none of them yet.
This is the third of four PRs porting the Lance read path (#70680). It is
stacked on the second, #70725, which is still in review: until that merges,
GitHub also shows its commits here; review only the last commit.
### What problem does this PR solve?
Issue Number: #70680
Related PR: #70725 (REST catalogs and OSS, the second PR; this builds on
it), #70681 (the plugin, the first PR), #70369 (ARRAY literals as element
lists, which `arrays_overlap` needs), #70372 (marks UDF calls, which this keeps
in Doris), #69581 (the BE Lance reader, which applies what this PR sends),
#65730, #66608, #66999, #67051, #67650, #68687 (the branch-4.x PRs this ports)
PR Type: Feature
Problem Summary:
On master, a Lance catalog scan leaves Lance all the work it could skip:
filters run in Doris after lance-c has read every row, lance-c never sees the
LIMIT, and COUNT(*) reads every fragment, while 4.1 and 4.2 push all three into
Lance. This PR ports 4.2's pushdown to the plugin. Conjuncts that Lance
evaluates exactly as Doris does go to lance-c as one Substrait filter and leave
the Doris plan; LIMIT reaches every range when no conjunct stays in Doris;
COUNT(*) without a filter comes from the manifest's row counts; and a pushed
filter that one scalar index can drive gets one range per index segment. For
example, on the regression table `doris.predicate_pushdown` (11 rows in 2
fragments), measured on this branch:
| Query | EXPLAIN of the Lance scan |
| --- | --- |
| `... WHERE int32_value = 10 LIMIT 1` |
`lancePushdownPredicate=(int32_value = 10)`, `lanceLimit=1`, no `PREDICATES:`
line |
| `... WHERE CAST(int32_value AS STRING) = '10' LIMIT 1` | `PREDICATES:
(CAST(int32_value[#4] AS text) = '10')`, no `lanceLimit`: Doris still filters,
as it filtered every conjunct before this PR |
| `SELECT count(*) ...` | `pushdown agg=COUNT (11)` |
Most of the diff is tests and their results (68% of the added lines). Query
results do not change.
Original authors: @zhangstar333, @Jay-ju and @Gabriel39.
### Details
- **Change map:** +8036 −66 in 115 files: production code +2568 −45, tests
+3290 −21 (FE unit tests +1786, regression suites +1177, fixture builders +320,
and 73 dataset files of 144 KiB, mostly binary), generated `.out` +2159, build
+19. The production code, all in the plugin except one session variable, in
reading order:
1. Predicate translation (+1,135): `LancePredicateConverter` (4.2's rules,
rewritten for `ConnectorExpression`), `LanceSubstraitSerializer` and
`lance_functions.yaml` (4.2's).
2. Index metadata (+493 −5): `LanceDatasetIndexDiscovery`,
`LanceIndexSegmentInfo` and `LanceIndexSegmentGroup` (4.2's, without the vector
and full-text parts), and `LanceIndexMetadata`, read once per statement through
`LanceCatalogClient` and `LanceStatementScope`.
3. Planning (+727): `LanceScalarIndexPlanner` (which index segment can
drive the filter), `LanceScanPlanner` and `LanceSplitBuilder` (ranges, COUNT
carriers, debug grouping), `LanceScanPlan` (what EXPLAIN prints).
4. Ranges and the provider (+197 −40): `LanceScanRange` (index segment,
`use_scalar_index`, limit, COUNT carrier) and `LanceScanPlanProvider` (which
conjuncts were pushed, the filter in `TLanceScanParams`, EXPLAIN).
5. fe-core (+16): the session variable `lance_fragments_per_split`,
declared there as the Fluss connector's `fluss_union_read_mode` is. Build:
`io.substrait:core` 0.40.0 (Apache-2.0, the version 4.x pins).
- **Scope:** Pushed, as on 4.2: comparisons (`<=>` included) between a
column and a literal, `[NOT] IN` lists without NULL, `IS [NOT] NULL`, `LIKE`
without a backslash, `starts_with`, `ends_with`, boolean columns, `AND` / `OR`
/ `NOT` whose operands all convert, and `array_contains` / `arrays_overlap` on
string arrays with non-NULL needles; on bool, integer, float32/64, decimal128,
utf8, date32 and zone-less timestamp(s/ms/us) columns. Only builtin calls are
translated: a Java or Python UDF with one of these names stays in Doris, as on
4.2. The rest stays in Doris under `PREDICATES:`, such as CASTs, comparisons
with NULL and other column types. Index segments: BTREE, BITMAP and LABEL_LIST,
for a positive predicate within lance-c's bounds (128 nodes, depth 32).
- **Design:** A conjunct is translated whole or not at all, and only exact
translations are reported pushed
(`ScanNodePropertiesResult.withPushdownTracking`, as the Elasticsearch
connector does), so the engine drops exactly those from the Doris residuals, as
4.2 did. Where the SPI forces a different shape from 4.2:
- CASTs stay in Doris (`supportsCastPredicatePushdown` returns false), as
on 4.2: the SPI default strips the CAST and would let Lance compare in the
column's own type.
- UDFs: 4.2 checked `TFunctionBinaryType.BUILTIN`; here the converter
refuses a `ConnectorFunctionCall` whose `isBuiltin()` is false (#70372).
- No plan state on the provider: the engine memoizes it by table handle,
which the snapshot pin replaces, so EXPLAIN (`getScanNodePropertiesResult`) and
the ranges (`planScan`) are planned deterministically from the same inputs.
- COUNT(*) carriers: 4.2 makes min(fragments, instances × backends), and
the SPI gives neither count, so 10,000 rows or more get one carrier per
fragment and fewer get one. Each carrier keeps its fragments for a BE that
falls back to scanning.
- Index metadata is read only when a segment could be planned (4.2 reads
it for every scan), so `DESC` and plain scans don't fail on an unreadable index.
- **Usage:** Nothing to enable:
```sql
EXPLAIN SELECT row_id FROM lance_s3.db.items WHERE row_id >= 2 AND label =
'a' LIMIT 10;
-- lanceFragmentGrouping=FRAGMENT
-- lanceLimit=10
-- lancePushdownPredicate=(row_id >= 2) AND (label = 'a')
-- and no PREDICATES: line, since nothing is left for Doris. When one
scalar index drives the
-- filter, lanceFragmentGrouping=INDEX_SEGMENT and
lanceGroupingIndex=<index> appear instead.
EXPLAIN SELECT COUNT(*) FROM lance_s3.db.items; -- pushdown agg=COUNT
(<rows>)
SET lance_fragments_per_split = 2; -- debugging: two fragments per range,
no index segments
```
- **Default behavior:** On for every Lance catalog scan;
`lance_fragments_per_split` defaults to 0 (automatic grouping) and rejects
negative values.
- **Compatibility:** No Thrift, Protobuf or persisted-metadata change, and
no connector SPI change, so the plugin API version stays as it is: the plugin
fills fields that master's BE has read since #69581 and 4.2's FE sends
(`lance_substrait_filter`; `limit`, `index_segment_uuids` and
`use_scalar_index` in `TLanceFileDesc`; the generic `table_level_row_count`).
`lance_fragments_per_split` keeps 4.2's name, default and checker, but is
visible: the engine hands connectors only visible session variables
(`VariableMgr.toMap`).
- **Follow-up plan:** PR 4 adds the session and table-access caches and the
profile counters; vector and full-text search, index inspection and the file
TVF follow, in the same issue. Later 4.2 changes to these files (#68813, #69625
and the search work) are ported with the next batch.
### Testing
- `fe-connector-lance` unit tests, 213, locally on this commit with
lance-core's JNI library loaded (none skipped): 4.2's converter cases on
`ConnectorExpression` input, its two Java-UDF cases included, plus new ones for
DECIMAL exactness, integer ranges and UDFs named like each translated builtin
(34); all 9 of 4.2's scalar-index planner cases; 13 of its scan-node cases (the
non-search planning ones; one now expects a COUNT carrier per fragment); its
two Thrift contract cases; and the provider's residual, LIMIT and COUNT rules.
- 9 mutations, each run against the unchanged tests: every one fails its
target tests (a UDF translated by name, a truncated DECIMAL, NULL needles
dropped, `<=>` without `IS NOT NULL`, an escaped LIKE pushed, no integer range
check, a LIMIT past a residual, every conjunct claimed pushed, COUNT despite a
filter).
- fe-core, 55 tests: `SessionVariablesTest` (the variable reaches connectors
and rejects negative values), `ExprToConnectorExpressionConverterTest`,
`ExprToConnectorFunctionCallTest` and `PluginDrivenScanNodeCastPredicateTest`.
The FE build ran checkstyle and the connector import gate.
- Regression, locally, with FE and an ASAN BE built from this commit, the
committed REST stub, and a host S3 endpoint that verifies SigV4 and serves the
committed fixture: the six new suites, PR 1's `test_lance_catalog_all_types`
and PR 2's `test_lance_rest_catalog` pass against the committed `.out` files (8
suites, 0 failed, 0 skipped). The new `.out` files are branch-4.2's results
block for block: int 176/176, scalar 178/178, optimize_count 10/10,
runtime_filter 1/1 (its two vector-search blocks come with search). In CI, the
External pipeline runs them.
### Notes for Reviewers
- Once #70725 merges, this branch is rebased onto master with only its last
commit.
- Master's plugin scan node prints Doris residuals as `PREDICATES:`, not
4.x's `predicates:`, so the 12 ported checks use that spelling (`notContains
"predicates:"` would always pass); `test_lance_runtime_filter_pushdown` keeps
only 4.2's non-search case and asserts `CONNECTOR: lance`.
- DECIMAL: 4.2 relies on `ExpressionCreator.decimal` throwing, but it
truncates to the column's scale (`12.505` becomes `12.50`), so a literal is
pushed only when the column's type holds it exactly; the unit test fails
without the check.
- `arrays_overlap` also requires a string element type: LARGEINT, IPv4 and
JSON elements arrive as Strings too (#70369).
- The fixture is committed instead of built by Spark: the `doris`
namespace's `__manifest`, `predicate_pushdown` (4.2's rows, two appends), and
4.2's `vs_ivf_pq_f32` (read by `test_lance_fragment_grouping`),
`multi_frag.lance` and `predicate_arrays/` unchanged.
`lance_build_predicate_pushdown.py` rebuilds or `--check`s it with 4.2's pinned
writer.
- The extension URI of `array_has` names the plugin's
`lance_functions.yaml`; lance-c resolves functions by name.
### Release note
The session variable `lance_fragments_per_split` (default 0) is a debugging
override for Lance catalog scans: a positive value groups that many fragments
into each scan range and turns index-segment ranges off. Unlike 4.1 and 4.2,
`SHOW VARIABLES` lists it.
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Partially verified: the regression suites ran
locally against a host S3 substitute and the REST stub, not yet in the External
pipeline
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, the fixture builder, this description
- Human review: The author reviewed part of the changes.
- Prompts: The author asked to port the branch-4.2 Lance FE read path to
master as a connector plugin in four PRs, of which this is the third; approved
handing ARRAY literals to connectors as element lists (#70369) and marking UDF
calls (#70372) in separate PRs first; had the regression fixture committed as a
dataset rather than built by Spark; and asked to build this PR on #70725
without waiting for it to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]