Gabriel39 commented on code in PR #66247:
URL: https://github.com/apache/doris/pull/66247#discussion_r3679350526
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -483,7 +498,51 @@ private static long nonNegative(long value) {
}
private static int currentAsyncReaderThreadCount() {
- return countThreadsByNamePrefix(ASYNC_READER_THREAD_NAME_PREFIX);
+ return ASYNC_READER_THREAD_COUNTER.get();
+ }
+
+ static String buildDebugSummary(int batchSize, int requiredFieldCount) {
+ return "Paimon JNI scanner configuration: batchSize=" + batchSize
+ + ", requiredFieldCount=" + requiredFieldCount;
+ }
+
+ static String[] requiredFields(Map<String, String> params) {
+ String encodedFields = params.get("required_fields_base64");
+ if (encodedFields == null) {
+ return splitParam(params.get("required_fields"), ",");
+ }
+ if (encodedFields.isEmpty()) {
Review Comment:
Fixed in 918e48adea1. Every outer schema token is now prefixed with `$`, so
`{}` remains an empty payload while `{\"\"}` becomes `$`. Both V1 and V2
producers use the shared encoder. Added C++ `{}`/`{\"\"}` coverage, Java
constructor/cardinality coverage, and P0 reads with an empty identifier under
both scanner routes.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -136,9 +137,12 @@ public Table getPaimonTable(NameMapping nameMapping,
String branch, String query
}
return executionAuthenticator.execute(() -> {
Table table = catalog.getTable(identifier);
- Map<String, String> tableOptions =
-
paimonProperties.getTableOptionsForCopy(table.options());
- return tableOptions.isEmpty() ? table :
table.copy(tableOptions);
+ Map<String, String> tableOptions =
paimonProperties.getTableOptionsForCopy();
+ Table effectiveTable = tableOptions.isEmpty() ? table :
table.copy(tableOptions);
+ // Physical table options bypass Doris property validation, so
validate the final
+ // merged view before Paimon can allocate batches or replace
its manifest executor.
+
PaimonReaderOptions.validateEffectiveTableOptions(effectiveTable.options());
Review Comment:
Fixed in 918e48adea1. Catalog loading now defers effective-option validation
until relation-scoped copies have been applied; `PaimonScanNode` validates
again at the last common planning boundary, so no-override and bypass paths
remain guarded. Added tests for safe catalog/relation overrides of unsafe
physical batch-size and manifest parallelism, plus rejection when the final
value is still unsafe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
