Gabriel39 commented on code in PR #66247:
URL: https://github.com/apache/doris/pull/66247#discussion_r3679059048
##########
be/src/format_v2/jni/jni_table_reader.cpp:
##########
@@ -517,6 +529,9 @@ void JniTableReader::_prepare_jni_scanner_schema() {
{column.transfer_type->create_column(), column.transfer_type,
column.java_name});
}
_scanner_params["required_fields"] = join(required_fields, ",");
+ // Preserve the legacy field during rolling upgrades, while new scanners
use independently
+ // encoded identifiers so commas and other legal identifier characters
cannot alter framing.
+ _scanner_params["required_fields_base64"] =
encode_jni_required_fields(required_fields);
Review Comment:
Fixed in b336eb4293f. The legacy PaimonJniReader now emits the same paired
required_fields_base64 and columns_types_base64 payload as FileScannerV2. The
P0 suite explicitly runs the quoted/nested projection once with
enable_file_scanner_v2=false and once with it enabled.
##########
be/src/format_v2/jni/jni_table_reader.cpp:
##########
@@ -517,6 +529,9 @@ void JniTableReader::_prepare_jni_scanner_schema() {
{column.transfer_type->create_column(), column.transfer_type,
column.java_name});
}
_scanner_params["required_fields"] = join(required_fields, ",");
+ // Preserve the legacy field during rolling upgrades, while new scanners
use independently
+ // encoded identifiers so commas and other legal identifier characters
cannot alter framing.
+ _scanner_params["required_fields_base64"] =
encode_jni_required_fields(required_fields);
_scanner_params["columns_types"] = join(column_types, "#");
Review Comment:
Fixed in b336eb4293f. Each complete type descriptor is now independently
Base64-framed, and nested STRUCT child names use a versioned Base64 token
inside the descriptor grammar. Unit/scanner/P0 coverage includes child names
containing #, comma, and colon.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -136,8 +136,7 @@ public Table getPaimonTable(NameMapping nameMapping, String
branch, String query
}
return executionAuthenticator.execute(() -> {
Table table = catalog.getTable(identifier);
- Map<String, String> tableOptions =
-
paimonProperties.getTableOptionsForCopy(table.options());
+ Map<String, String> tableOptions =
paimonProperties.getTableOptionsForCopy();
Review Comment:
Fixed in b336eb4293f. PaimonExternalCatalog validates the final
physical-plus-catalog table options before planning/serialization. The P0
fixture creates physical tables with read.batch-size=0 and
scan.manifest.parallelism=0 under a catalog with no overrides and verifies both
are rejected.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java:
##########
@@ -656,11 +656,17 @@ public void replayAlterCatalogProps(CatalogLog log,
Map<String, String> oldPrope
if (!isReplay) {
try {
((ExternalCatalog) catalog).checkProperties();
- } catch (DdlException ddlException) {
+ } catch (Exception validationException) {
Review Comment:
Fixed in b336eb4293f. Paimon ALTER validation now builds and validates a
detached CatalogProperty candidate before the live map is changed. A
latch-controlled CatalogMgr test overlaps validation with first initialization
and verifies that initialization observes only the old property map after the
candidate is rejected.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java:
##########
@@ -188,44 +172,14 @@ private Map<String, String> extractTableOptions() {
}
private void validateTableOption(String key, String value) {
- ConfigOption<?> option = SUPPORTED_TABLE_OPTIONS.find(key);
- if (option == null) {
- throw new IllegalArgumentException("Unsupported Paimon table
option '" + key
- + "' for the bundled Paimon version");
- }
-
try {
- new Options(Collections.singletonMap(key, value)).get(option);
+ PaimonReaderOptions.validate(key, value);
Review Comment:
Fixed in b336eb4293f. New CREATE and changed ALTER table options remain
strictly validated, while lazy reconstruction filters unsupported or invalid
legacy options without deleting the persisted raw properties. Image round-trip
and edit-log replay tests verify that a legacy write.batch-size entry remains
loadable but is not applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
