This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
omit a2a6f696bca [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit b9555445177 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit e7331bc449d [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit d63ac1539ba [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit 017dc7ac6f7 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit a288c4ff75d [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit ef1af161ac2 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit f730cdb925c [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit 89ae658f8f0 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 4aa2a58d010 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit 5f2bef56b1e [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit 0662262e039 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit f73e549e186 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit 3044e390313 [refactor](catalog) Catalog spi 11 hive (#65473)
omit 2a621c7b1f2 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit d14889d4c05 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit 6e44c045b7e [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 28f64a6d3d9 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 817bd0f1299 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit ad652613226 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 3c290dafbd3 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit 631ebfc004d [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit 233d09feef8 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit c3208830f2f [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 76798cdfe87 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit c7fb8b41d1f [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit c5c3abf0462 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 917874e933b [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit a7adebbb593 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit 1738555ca07 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit d92bff215f5 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 09bbf92b7c4 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 207df0b5aab [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 2706c7eb2d6 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 0cf79390f00 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit ae6ccc6992a update P5 handoff and fix compile issue
omit 623c808a071 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit e9d3779c0ef [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit ae92c0e7fda [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 63989d46680 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit d0ae3cb7d3d [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit 1696ff77281 [doc](connector) add project tracking system for catalog
SPI migration
add 1b0217a7c83 [fix](iceberg) Honor partial name mappings for legacy
files (#65784)
add bf55619038c [doc](connector) add project tracking system for catalog
SPI migration
add e498bbacc90 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add f69a97b57bb [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 857fd021316 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add 9cff003555b [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 0aa26cf6538 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add ecfa5bb5bfb update P5 handoff and fix compile issue
add 98dfd62fe28 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add c1bff95f7fc [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add 8b3c9b5ba7d [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 0454fc73a7f [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 30b21e8fa3d [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 9c0eb9857b7 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 05b1e249fb8 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add cfd7c00addf [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add e474c9c7f3a [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 499e123e156 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 337a735c184 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 577a19a0ef1 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 2544b1bb9e2 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 590c8deec33 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 7dc3f4aa35e [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add c69921195cd [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 69f5ce05491 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 51c91a90a35 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 0fcd85f6a38 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 4816b0e6771 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add fdacbdc9477 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 6e521aa64b2 [refactor](catalog) Catalog spi 11 hive (#65473)
add fb2c7becd1e [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add e796f8be24f [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add 38dec3d48de [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add 521dd776fee [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 1928d21af13 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add 0b3c3c2cb2a [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add ddacf2a5578 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add fa29dd5aef0 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add f8f5f3711f9 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add 67dd69637e3 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add 0b4f72582e7 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add 5c744a5898c [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 019e7ae6954 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add b505de3d725 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a2a6f696bca)
\
N -- N -- N refs/heads/branch-catalog-spi (b505de3d725)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/exec/scan/access_path_parser.cpp | 101 ++-
be/src/exec/scan/access_path_parser.h | 6 +-
be/src/exec/scan/file_scanner_v2.cpp | 7 +-
be/src/format/orc/vorc_reader.cpp | 53 +-
be/src/format/parquet/vparquet_column_reader.cpp | 50 +-
.../iceberg_position_delete_sys_table_reader.cpp | 86 ++-
be/src/format/table/iceberg_reader.cpp | 105 ++-
be/src/format/table/iceberg_reader_mixin.h | 150 ++++-
.../table/iceberg_scan_semantics.h} | 13 +-
be/src/format/table/table_schema_change_helper.cpp | 284 +++++++--
be/src/format/table/table_schema_change_helper.h | 59 +-
be/src/format_v2/column_data.h | 6 +
be/src/format_v2/column_mapper.cpp | 127 +++-
be/src/format_v2/column_mapper.h | 4 +
.../iceberg_position_delete_sys_table_reader.cpp | 31 +-
be/src/format_v2/table/iceberg_reader.cpp | 15 +-
be/src/format_v2/table/iceberg_reader.h | 36 +-
be/src/format_v2/table/iceberg_schema_utils.h | 53 ++
be/src/format_v2/table/schema_history_util.cpp | 2 +
be/src/format_v2/table_reader.cpp | 138 +++-
be/src/format_v2/table_reader.h | 7 +-
be/test/exec/scan/access_path_parser_test.cpp | 100 ++-
...eberg_position_delete_sys_table_reader_test.cpp | 411 ++++++++++++
.../format/table/iceberg/iceberg_reader_test.cpp | 701 ++++++++++++++++++++-
.../table/table_schema_change_helper_test.cpp | 456 ++++++++++++++
be/test/format_v2/column_mapper_test.cpp | 75 +++
be/test/format_v2/table/iceberg_reader_test.cpp | 537 +++++++++++++++-
be/test/format_v2/table_reader_test.cpp | 236 ++++++-
.../connector/iceberg/IcebergScanPlanProvider.java | 12 +
.../connector/iceberg/IcebergSchemaUtils.java | 87 ++-
.../iceberg/IcebergScanPlanProviderTest.java | 18 +
.../connector/iceberg/IcebergSchemaUtilsTest.java | 69 +-
gensrc/thrift/ExternalTableSchema.thrift | 5 +-
gensrc/thrift/PlanNodes.thrift | 4 +
34 files changed, 3703 insertions(+), 341 deletions(-)
copy be/src/{io/fs/path.h => format/table/iceberg_scan_semantics.h} (61%)
create mode 100644 be/src/format_v2/table/iceberg_schema_utils.h
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
