morningman commented on code in PR #70827:
URL: https://github.com/apache/doris/pull/70827#discussion_r4657495867
##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceScanPlanProvider.java:
##########
@@ -85,30 +101,31 @@ final class LanceScanPlanProvider implements
ConnectorScanPlanProvider {
@Override
public List<ConnectorScanRange> planScan(ConnectorSession session,
ConnectorScanRequest request) {
Review Comment:
Agreed on how it works today. It isn't planned in this port (#70680), and
for Lance it takes more than the two methods:
1. Not new, and not dropped by the port: `planScan` has been synchronous
since #70681, and the `LanceScanNode` of 4.1 and 4.2 has no batch mode either
(it doesn't override `isBatchMode()`).
2. Streaming would only defer the range objects, not the per-fragment work.
The fragment list is complete before any split is planned: when the statement
pins the version,
[`readMetadata`](https://github.com/apache/doris/blob/c4d996f50697756c557c778eb9e90c4585081de7/fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceCatalogClient.java#L205-L211)
walks `Dataset.getFragments()`, which in lance-core 12.0.0 returns the whole
list from native code. The index-segment plan needs every fragment up front as
well. Measured on the plugin's planner (FRAGMENT grouping, JDK 17, warm
medians): 0.9 ms for 10k fragments, and 8.9 ms plus about 17 MB of
`LanceScanRange`s for 100k. The engine's split and thrift objects come on top
of that.
3. With today's hooks it would cost this PR's LIMIT pushdown.
[`prepareBatchScan`](https://github.com/apache/doris/blob/c4d996f50697756c557c778eb9e90c4585081de7/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java#L2352-L2365)
builds the batch request without LIMIT or COUNT(*), and
[`streamingSplitEstimate`](https://github.com/apache/doris/blob/c4d996f50697756c557c778eb9e90c4585081de7/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java#L436-L440)
doesn't see the limit, so a connector can't keep LIMIT scans synchronous.
Since `num_files_in_batch_mode` [defaults to
1](https://github.com/apache/doris/blob/c4d996f50697756c557c778eb9e90c4585081de7/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java#L2663-L2668),
an Iceberg-style gate would stream almost every Lance scan. lance-c would then
lose the per-range limit, while EXPLAIN would still print `lanceLimit=`,
because the engine
hands the limit to `appendExplainInfo` [either
way](https://github.com/apache/doris/blob/c4d996f50697756c557c778eb9e90c4585081de7/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java#L769).
COUNT(*) could stay synchronous, as Iceberg does.
It becomes worth doing once a real dataset shows split materialization,
rather than the manifest read, as the cost, and once a batch request can carry
the limit. Could the engine pass the limit to `streamingSplitEstimate` and into
the batch request? With that, Lance could stream large scans and keep the
pushdown.
##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceScanPlanProvider.java:
##########
@@ -85,30 +101,31 @@ final class LanceScanPlanProvider implements
ConnectorScanPlanProvider {
@Override
public List<ConnectorScanRange> planScan(ConnectorSession session,
ConnectorScanRequest request) {
- LanceTableMetadata metadata = metadata(session,
lanceHandle(request.getTableHandle()));
- List<LanceFragmentInfo> fragments = metadata.getFragments();
- Set<Long> seen = new HashSet<>();
- long targetSplitSize = 1;
- for (LanceFragmentInfo fragment : fragments) {
- if (!seen.add(fragment.getId())) {
- throw new DorisConnectorException("Duplicate Lance fragment id
" + fragment.getId()
- + " at dataset version " + metadata.getVersion());
- }
- // Weights are proportional to the largest range of this plan.
- targetSplitSize = Math.max(targetSplitSize,
fragment.getSchedulingWeight());
+ LanceTableHandle handle = lanceHandle(request.getTableHandle());
+ LanceTableMetadata metadata = metadata(session, handle);
+ List<ConnectorExpression> conjuncts =
LancePredicateConverter.conjuncts(request.getFilter());
+ // Any filter, pushed or not, makes the version's row count larger
than the result.
+ if (request.isCountPushdown() && conjuncts.isEmpty()) {
+ return LanceScanPlanner.countRanges(metadata);
}
- List<ConnectorScanRange> ranges = new ArrayList<>(fragments.size());
- for (LanceFragmentInfo fragment : fragments) {
- ranges.add(new LanceScanRange(metadata.getDatasetUri(),
metadata.getVersion(),
- Collections.singletonList(fragment.getId()),
fragment.getSchedulingWeight(), targetSplitSize));
- }
- return ranges;
+ LancePredicateConverter converter = new
LancePredicateConverter(metadata.getSchema());
+ LancePredicateConverter.ConversionResult pushdown =
converter.convert(conjuncts);
Review Comment:
Right, `planScan` serializes the filter a second time. That is deliberate:
the provider keeps no state, so the scan properties and the ranges are planned
from the same inputs (see Design in the description). The cost is small enough
that I'd keep it:
- `LanceSubstraitSerializer.serialize`, measured per call on JDK 17 (warm):
2.2 µs for two comparisons on a 20-column table, 9.7 µs on 200 columns, 51 µs
with a 1000-value IN, and 0.54 ms for the largest case I tried, a 10k-value IN
on 200 columns. It grows with the table width and the filter, not with the
fragment count, and runs once per scan node. Reading the table's metadata takes
milliseconds per statement (6 ms in the profile in #70830's description).
- A statement-scoped result would need the filter in its key. A self-join or
UNION ALL of one table at one version plans two scans with different filters,
and handing one scan the other's pushed set would return wrong rows. Caching
the whole plan would also look up the index metadata from inside a scope
loader, which can fail with ConcurrentHashMap's "Recursive update" (we hit that
while building #70830).
If you'd still like it gone, a lazy `getSubstraitFilter()` in
`ConversionResult` is about ten lines. I can put it in #70830, which already
reworks `planScan` and reports its time in the profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]