github-actions[bot] commented on code in PR #67186:
URL: https://github.com/apache/doris/pull/67186#discussion_r3921054087
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java:
##########
@@ -1918,51 +2006,81 @@ public void startSplit(int numBackends) {
final List<String> allPartitions =
new
ArrayList<>(selectedPartitions.selectedPartitions.keySet());
final int batchSize = sessionVariable.getNumPartitionsInBatchMode();
+ SummaryProfile batchSummaryProfile =
SummaryProfile.getSummaryProfile(ConnectContext.get());
+ final RuntimeProfile batchExecutionSummary = batchSummaryProfile ==
null
+ ? null : batchSummaryProfile.getExecutionSummary();
Executor scheduleExecutor =
Env.getCurrentEnv().getExtMetaCacheMgr().getScheduleExecutor();
AtomicReference<UserException> batchException = new
AtomicReference<>(null);
- AtomicInteger numFinishedPartitions = new AtomicInteger(0);
-
- CompletableFuture.runAsync(() -> {
Review Comment:
[P1] Close the connector statement scope for batch scans.
`FileQueryScanNode` enters this path through `SplitAssignment.init()` ->
`startSplit()`, so it bypasses `getSplits()`, the only place that currently
registers `statementScope::closeAll`. The session captured here can own Iceberg
`TableLease`/`TrackedTable` resources, but Arrow Flight sets
`returnResultFromLocal=false` and `StatementContext.close()` deliberately skips
its fallback close, relying on the query-finish callback. Because this batch
path never registers that callback, each deferred Flight batch query can leave
its scope and leases open. Please factor the callback registration into a
helper and invoke it before dispatch in both batch flavors, with a lifecycle
test that verifies closure after coordinator unregister.
##########
fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClientConfig.java:
##########
@@ -128,10 +135,32 @@ public String getMetastoreType() {
return properties.getOrDefault(METASTORE_TYPE_KEY, METASTORE_TYPE_HMS);
}
+ public int getPartitionBatchSize() {
+ return partitionBatchSize;
+ }
+
+ private static int parsePositiveInt(Map<String, String> properties, String
key, int defaultValue) {
+ String value = properties.get(key);
+ if (value == null) {
+ return defaultValue;
+ }
+ int parsed;
+ try {
Review Comment:
[P2] Normalize the raw batch-size property before parsing. The catalog
binder trims values before binding this field, but both
`HiveCatalogProperties.of` and `HudiCatalogProperties.of` then pass the
original map to this constructor, where `Integer.parseInt(value)` is called
without trimming. A quoted SQL property such as
`hive.hms_partitions_batch_size_per_rpc = ' 5000 '` therefore binds
successfully and then throws while creating/replaying the catalog. Please trim
(or pass the typed field) and add Hive/Hudi whitespace tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
