This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 1aa5ae9597e [fix](catalog) port #66008's paimon-cpp removal to the
paimon connector scan path
omit 0d88b8dec20 [fix](iceberg) dedupe partition columns so a
multi-transform spec cannot crash partition pruning
omit b68cf015a57 [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
omit 827dc4f0ca6 [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
omit 710ed241df6 [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
omit 5d49b866d6a [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
omit 4417b12c338 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
omit f2d579fbda4 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
omit 4b918b025db [fix](be) fail the query instead of aborting BE on a
missing projected column
omit 19a90a7dba1 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
omit ead8a9924cd [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
omit f66d3be2f7b [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
omit 4dc44f71292 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
omit 645a18e86b0 [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
omit 6a6d2f4a06a [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
omit 0a2568a4aa4 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
omit 336556289df [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
omit 989a10bc96d [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
omit 2b8f84b5c29 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
omit 231f8fedec3 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
omit 8352e6b02b6 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit 13638182a41 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit 2ee432deb94 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit 0ef0fbeab7d [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit da9ce090f3b [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit 20de518986d [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit e027acdb600 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit 3fd609552c0 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit ed2206c2c69 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 16c3278e93f [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit 7932e374909 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit 02284e682c7 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 6117bcb22bb [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit 53766ceb32b [refactor](catalog) Catalog spi 11 hive (#65473)
omit 48c3e6966a5 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit 7b09e195ee7 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit d8b105aee3d [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 96fd92c1acd [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 7516fe68a43 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit 0e75080be3e [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 25c9199706c [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit e97aad2f57f [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit e55d3a68e77 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit c66fcc06749 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 335012d1e16 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 4fd80cd6e67 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit d64ad6d85f7 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 9aaf6686e7b [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit ff7dd1a945d [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit ed05170c7e7 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit a106157a2ab [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 76ecd62be3a [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit e5de5f00873 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit ff3088dfc7a [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit b156b0f304f [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit 090596c4844 update P5 handoff and fix compile issue
omit 76e279bbcc7 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit de930d575ce [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit 5c658ea072e [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 8f230a093ae [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 2cf57f79638 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit a6cb29ab2ba [doc](connector) add project tracking system for catalog
SPI migration
add dcc109a3923 [chore](ci) add storage/rowset/tablet code owners (#66053)
add cb56b23822a [refactor](storage) extract RowKeyEncoder: dedup key
encoding across both segment writers (#65492)
add 0c35a6e6d8b [feature](plugin) Add information_schema.plugins for
kernel plugin metadata (#65644)
add 678a4c070ca [fix](jdbc) Harden JDBC driver URL validation and remove
file-upload HTTP API (#65987)
add 5b3ac63f8b4 [improvement](parquet) Make V2 column initialization lazy
(#66073)
add 29e98c098af [doc](connector) add project tracking system for catalog
SPI migration
add e7a8409b2d5 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add 9d533cf9d72 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add ff80ee37fcb [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add bba524d6ef0 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 3df0ec8d2ab [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 116ea6d7f46 update P5 handoff and fix compile issue
add bddddbd4d50 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 2498943fe2f [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add a92320fa4e4 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add a953ddd71d5 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 863a7e3016c [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add a26d2e6d325 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add fa11340244b [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add 2195048a827 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add 38bc1a9f580 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 25a92be3f8a [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 0e33af61629 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 20b186263eb [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 7cf5d8c2701 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add f62031a4510 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 78945041686 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add 128c1d4a6a4 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 8ddd852117c [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add afbc33471d8 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 83c8eb0a397 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 49e4a6e67d0 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 909a8438b38 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add a901b285253 [refactor](catalog) Catalog spi 11 hive (#65473)
add ba1b51933d6 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 931642e4ed6 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add f9c6fa86cae [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add 515cab2e104 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 4ea88483636 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add 385ad651e56 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add ee64bada109 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 09b31a13c29 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add ce8805db052 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add 8a55a725022 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add d7c4c577638 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add ece8eb53f52 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 815ca38973e [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add 97d7e179e4e [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add e65d0f6feca [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add bc3637d5f58 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add 6e850ad8cd7 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add 553d74b8666 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add f15bda5c8d9 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
add eaa4680b35c [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
add ef874a130e6 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
add 0a1696314a5 [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
add 5c081c7d3c2 [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
add 17efb90f377 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
add 4193cf645fb [fix](be) fail the query instead of aborting BE on a
missing projected column
add 1bc292b02a5 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
add ea47ac1664a [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
add eac42bdacd3 [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
add a122189799c [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
add 8bd8d058b03 [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
add 5243176d83f [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
add ffe6c96cab7 [fix](iceberg) dedupe partition columns so a
multi-transform spec cannot crash partition pruning
add 7837c5698df [fix](catalog) port #66008's paimon-cpp removal to the
paimon connector scan path
add ceb33843d4b [chore](handoff) record the 2026-07-27 rebase onto
5b3ac63f8b4
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1aa5ae9597e)
\
N -- N -- N refs/heads/branch-catalog-spi (ceb33843d4b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/CODEOWNERS | 14 +-
.../parquet/reader/native/column_chunk_reader.cpp | 60 +-
.../parquet/reader/native/column_chunk_reader.h | 7 +-
.../parquet/reader/native/column_reader.cpp | 25 +-
.../iceberg_position_delete_sys_table_reader.cpp | 10 +-
...s_scanner.cpp => schema_extensions_scanner.cpp} | 94 +-
...pings_scanner.h => schema_extensions_scanner.h} | 19 +-
be/src/information_schema/schema_scanner.cpp | 3 +
be/src/runtime/runtime_profile.cpp | 17 +
be/src/runtime/runtime_profile.h | 5 +
be/src/storage/key/row_key_encoder.cpp | 155 +++
be/src/storage/key/row_key_encoder.h | 93 ++
be/src/storage/partial_update_info.cpp | 11 +-
be/src/storage/segment/segment_writer.cpp | 153 +--
be/src/storage/segment/segment_writer.h | 30 +-
be/src/storage/segment/vertical_segment_writer.cpp | 168 +--
be/src/storage/segment/vertical_segment_writer.h | 26 +-
.../schema_extensions_scanner_test.cpp | 126 +++
.../storage/key/row_key_encoder_all_key_types.out | 410 +++++++
be/test/format_v2/parquet/native_decoder_test.cpp | 506 ++++++++-
be/test/runtime/runtime_profile_test.cpp | 31 +
be/test/storage/key/row_key_encoder_test.cpp | 1138 ++++++++++++++++++++
be/test/storage/segment/test_segment_writer.h | 2 +-
.../handler/AuthenticationPluginManager.java | 40 +-
.../handler/AuthenticationPluginManagerTest.java | 16 +
.../main/java/org/apache/doris/common/Config.java | 24 +-
.../java/org/apache/doris/common/ConfigTest.java | 26 +-
.../connector/jdbc/JdbcConnectorProvider.java | 5 +
.../doris/connector/jdbc/JdbcDorisConnector.java | 63 ++
.../jdbc/JdbcConnectorProviderValidateTest.java | 31 +-
.../jdbc/JdbcDriverUrlSecurityRuleTest.java | 92 ++
.../org/apache/doris/analysis/SchemaTableType.java | 3 +-
.../org/apache/doris/catalog/JdbcResource.java | 133 ++-
.../java/org/apache/doris/catalog/SchemaTable.java | 8 +
.../doris/connector/ConnectorPluginManager.java | 35 +-
.../apache/doris/fs/FileSystemPluginManager.java | 54 +-
.../org/apache/doris/httpv2/rest/UploadAction.java | 318 ------
.../apache/doris/httpv2/util/LoadSubmitter.java | 178 ---
.../org/apache/doris/httpv2/util/TmpFileMgr.java | 307 ------
.../nereids/lineage/LineageEventProcessor.java | 18 +
.../doris/tablefunction/MetadataGenerator.java | 39 +
.../org/apache/doris/catalog/JdbcResourceTest.java | 166 +++
fe/fe-extension-loader/pom.xml | 10 +
.../loader/DirectoryPluginRuntimeManager.java | 78 +-
.../doris/extension/loader/ManifestVersions.java | 92 ++
.../doris/extension/loader/PluginHandle.java | 20 +
.../apache/doris/extension/loader/PluginNames.java | 62 ++
.../doris/extension/loader/PluginRegistry.java | 219 ++++
.../DirectoryPluginRuntimeManagerMetadataTest.java | 143 +++
.../doris/extension/loader/PluginRegistryTest.java | 145 +++
.../testplugins/BadNameTestPluginFactory.java} | 19 +-
.../testplugins/MetadataTestPluginFactory.java} | 30 +-
.../apache/doris/extension/spi/PluginFactory.java | 11 +
gensrc/thrift/Descriptors.thrift | 1 +
gensrc/thrift/FrontendService.thrift | 1 +
plan-doc/HANDOFF.md | 55 +-
.../schema_table/test_extensions_schema.groovy | 81 ++
57 files changed, 4314 insertions(+), 1312 deletions(-)
copy be/src/information_schema/{schema_role_mappings_scanner.cpp =>
schema_extensions_scanner.cpp} (53%)
copy be/src/information_schema/{schema_role_mappings_scanner.h =>
schema_extensions_scanner.h} (71%)
create mode 100644 be/src/storage/key/row_key_encoder.cpp
create mode 100644 be/src/storage/key/row_key_encoder.h
create mode 100644
be/test/exec/schema_scanner/schema_extensions_scanner_test.cpp
create mode 100644
be/test/expected_result/storage/key/row_key_encoder_all_key_types.out
create mode 100644 be/test/storage/key/row_key_encoder_test.cpp
create mode 100644
fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDriverUrlSecurityRuleTest.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/UploadAction.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/httpv2/util/LoadSubmitter.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/httpv2/util/TmpFileMgr.java
create mode 100644
fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/ManifestVersions.java
create mode 100644
fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/PluginNames.java
create mode 100644
fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/PluginRegistry.java
create mode 100644
fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerMetadataTest.java
create mode 100644
fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/PluginRegistryTest.java
copy fe/{fe-type/src/main/java/org/apache/doris/catalog/AnyElementType.java =>
fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/BadNameTestPluginFactory.java}
(66%)
copy fe/{fe-core/src/main/java/org/apache/doris/nereids/types/StringType.java
=>
fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/MetadataTestPluginFactory.java}
(62%)
create mode 100644
regression-test/suites/query_p0/schema_table/test_extensions_schema.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
