924060929 commented on code in PR #70827:
URL: https://github.com/apache/doris/pull/70827#discussion_r4656423962


##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceScanPlanProvider.java:
##########
@@ -85,30 +101,31 @@ final class LanceScanPlanProvider implements 
ConnectorScanPlanProvider {
 
     @Override
     public List<ConnectorScanRange> planScan(ConnectorSession session, 
ConnectorScanRequest request) {
-        LanceTableMetadata metadata = metadata(session, 
lanceHandle(request.getTableHandle()));
-        List<LanceFragmentInfo> fragments = metadata.getFragments();
-        Set<Long> seen = new HashSet<>();
-        long targetSplitSize = 1;
-        for (LanceFragmentInfo fragment : fragments) {
-            if (!seen.add(fragment.getId())) {
-                throw new DorisConnectorException("Duplicate Lance fragment id 
" + fragment.getId()
-                        + " at dataset version " + metadata.getVersion());
-            }
-            // Weights are proportional to the largest range of this plan.
-            targetSplitSize = Math.max(targetSplitSize, 
fragment.getSchedulingWeight());
+        LanceTableHandle handle = lanceHandle(request.getTableHandle());
+        LanceTableMetadata metadata = metadata(session, handle);
+        List<ConnectorExpression> conjuncts = 
LancePredicateConverter.conjuncts(request.getFilter());
+        // Any filter, pushed or not, makes the version's row count larger 
than the result.
+        if (request.isCountPushdown() && conjuncts.isEmpty()) {
+            return LanceScanPlanner.countRanges(metadata);
         }
-        List<ConnectorScanRange> ranges = new ArrayList<>(fragments.size());
-        for (LanceFragmentInfo fragment : fragments) {
-            ranges.add(new LanceScanRange(metadata.getDatasetUri(), 
metadata.getVersion(),
-                    Collections.singletonList(fragment.getId()), 
fragment.getSchedulingWeight(), targetSplitSize));
-        }
-        return ranges;
+        LancePredicateConverter converter = new 
LancePredicateConverter(metadata.getSchema());
+        LancePredicateConverter.ConversionResult pushdown = 
converter.convert(conjuncts);

Review Comment:
   Minor: `planScan()` calls `converter.convert(conjuncts)` again, which 
re-serializes the Substrait protobuf `byte[]` even though `planScan()` only 
needs `pushedConjuncts` and `allPushed()` (and `getScanNodePropertiesResult()` 
already serialized it). We could skip Substrait serialization on the 
`planScan()` path or reuse the statement-scoped planning result.



##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceScanPlanProvider.java:
##########
@@ -85,30 +101,31 @@ final class LanceScanPlanProvider implements 
ConnectorScanPlanProvider {
 
     @Override
     public List<ConnectorScanRange> planScan(ConnectorSession session, 
ConnectorScanRequest request) {

Review Comment:
   `LanceScanPlanProvider` currently only implements synchronous `planScan()` 
(single-threaded, materializing all `LanceScanRange`s into FE memory at once) 
and does not implement batch/streaming split mode (`streamingSplitEstimate` + 
`streamSplits`, like `IcebergScanPlanProvider`).
   
   Although Lance manifests avoid Hive's per-partition RPC I/O, a table with a 
large number of uncompacted fragments (e.g. tens/hundreds of thousands, 1 
fragment per split by default) will still materialize all splits synchronously 
on the FE planning thread and hold them in memory. Are there plans to support 
streaming batch mode (`num_files_in_batch_mode` / 
`enable_external_table_batch_mode`) here or in a follow-up?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to