morningman opened a new pull request, #70369:
URL: https://github.com/apache/doris/pull/70369
### Background
A catalog connector learns which filters it may push down from an
engine-neutral expression tree (`ConnectorExpression`) that fe-core's
`ExprToConnectorExpressionConverter` builds from the planner's conjuncts. A
literal in that tree carries a plain Java value whose shape the SPI documents
per Doris type (Rule 4 in `connector/spi/pushdown/package-info.java`): `Long`
for integers, `LocalDate` for dates, and so on. Types outside that table, ARRAY
among them, arrive as the literal's display text.
### What problem does this PR solve?
Issue Number: None
Related PR: #68687 (the Lance array-membership pushdown on branch-4.2, whose
master port needs the elements)
PR Type: Enhancement
Problem Summary:
An ARRAY literal reaches a connector as text such as `[red, a, b]`, with
unquoted elements, so `['a, b']` and `['a', 'b']` cannot be told apart. The
Lance connector needs the elements to push down `arrays_overlap`, which the
planner makes from three or more ORed `array_contains`. This PR makes the value
an unmodifiable `List` of element values, each in the shape the SPI gives its
element type. An audit of the connectors that read literal values found two
that would treat the list as text: JDBC would write it into remote SQL
unquoted, and Elasticsearch already pushes array comparisons as a term query on
the text. Both now leave such predicates to Doris.
### Details
- **Before and after:** For `['red', 'a, b', NULL]`,
`ConnectorLiteral#getValue()` was the String `[red, a, b, NULL]` and is now the
list `["red", "a, b", null]`; nested arrays are nested lists, and a NULL array
is still a NULL literal. An element has the value it would have as a literal of
its own (a LARGEINT element is still a decimal String). JDBC: `tags = ['a',
'b']`, or on MySQL `arrays_overlap(tags, ['a', 'b'])`, is no longer sent to the
source, where it went as the string `'[a, b]'`. Elasticsearch (docvalue mode):
`tags = ['red', 'blue']` is no longer pushed as `{"term":{"tags":"[red,
blue]"}}`.
- **Configs and defaults:** None.
- **Impact on existing users:** JDBC catalogs filter such rows in Doris
instead of the source, and the query no longer pushes LIMIT. Elasticsearch
drops a pushed comparison from what Doris evaluates, and a term query matches
any one value of a multi-valued field rather than the whole array, so `=` could
lose rows and `!=` could add some; this follows from the term query's semantics
and was not reproduced on a live cluster. ADBC, Iceberg, Paimon, MaxCompute and
Trino already refuse ARRAY literals, and Hive and Hudi partition pruning never
sees one. No SPI type or method changes, so the connector plugin API version
stays.
### Testing
All run locally on this commit's code:
- New `ExprToConnectorExpressionConverterTest` (fe-core): elements with
commas and quotes, every scalar shape, NULL elements, nested and empty arrays,
an unmodifiable list, a NULL array. 4 tests pass; with the new branch removed
(master's behavior) all 4 fail.
- `JdbcQueryBuilderTest#testArrayLiteralsRemainLocalWithoutLimit` (MySQL,
PostgreSQL, ClickHouse, Trino): no WHERE or LIMIT for a comparison, NOT or
`arrays_overlap` with an ARRAY literal; an AND keeps its other conjunct.
Without the new check the builder emits ``WHERE (`tags` = [a, b, x') OR
('1'='1]) LIMIT 1``.
- `EsQueryDslBuilderTest#arrayLiteralComparisonsRemainLocal`: `=`, `!=`, `<`
on either side stay in Doris; in an AND the other conjunct is still pushed.
- Six mutations (text instead of the list, a modifiable list, elements as
text, NULL elements dropped, each connector check removed), each against its
test class beside an unmutated control: every one fails a new test.
- Existing suites pass: fe-connector-spi (149, with
`ConnectorPluginSurfaceTest`), fe-connector-es (165), fe-connector-jdbc (276),
and in fe-core the converters that route through this one plus
`PluginDrivenScanNode*Test` (317).
- End to end, on a local cluster whose FE also carries the Lance connector
PR that builds on this one: `test_lance_array_predicate_pushdown` pushes
`arrays_overlap(labels, [...])` into Lance, up to the 64-needle index budget,
and its BE profile shows the scalar index segments searched.
### Notes for Reviewers
- `NereidsToConnectorExpressionConverter` (DELETE / UPDATE / MERGE) and
`UnboundExpressionToConnectorPredicateConverter` (`EXECUTE ... WHERE`) convert
literals through `ExprToConnectorExpressionConverter`, so all three paths get
the list.
- `convertLiteral` and `convertDateLiteral` now declare `ConnectorLiteral`,
which they always returned, so the element loop needs no cast.
### Release note
For JDBC and Elasticsearch catalogs, a filter that compares a column with an
ARRAY constant, such as `tags = ['a', 'b']`, and for JDBC a function call that
takes one, is now evaluated by Doris instead of being sent to the source.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: not run against live JDBC
sources or Elasticsearch
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, this description
- Human review: The author reviewed part of the changes.
- Prompts: While porting the Lance pushdown from branch-4.2, the agent
proposed handing ARRAY literals to connectors as element lists; the author
approved that plan: change the value, document it in the SPI, audit every
connector first, and send it as its own PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
