This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 568c4bb4571 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit 777a61671ab [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit 1ea735ff0a5 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit e697837760d [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit 11f4deaa508 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit 707f438f81e [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit 031ddd5a804 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit f61d42c2247 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 2535c250f40 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit 677a2326745 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit f4529a9133c [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 902eec59ef0 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit 3593684715f [refactor](catalog) Catalog spi 11 hive (#65473)
omit ad3d6f73b3a [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit ab1762d04fd [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit 23cf09a4fcb [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 0adf8287dfa [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit fb80955876f [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit 349effe9a9c [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 68ac95c46ba [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit a788e36a30d [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit f4fb8d3d708 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 83578743c19 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 5d8bcce16b2 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 9b77fad0ff8 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit 734610cc2cb [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 62da3ee7157 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit 6f970e40c3b [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit 9ea01f46038 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 22e3aa10935 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 1eceb20b70e [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 290911fbc3c [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 186f0255631 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 494183213c8 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit 26917f3a62a update P5 handoff and fix compile issue
omit d3c517b15d1 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit 1f7a53bb67a [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit d78b6e0a2c8 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 11b12397955 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 0dbbe4b0b9c [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit a7951aac9fe [doc](connector) add project tracking system for catalog
SPI migration
add e51099a9389 [fix](fe) Treat json_extract_no_quotes as
json_extract_string alias (#65380)
add f5995f402b3 [fix](load) support compute_group in FE stream load
routing and planning (#65571)
add ae88ed43c3f [fix](regression) Isolate non-catalog Kerberos test
database (#65825)
add d3b9f5bdbc5 [fix](ci) adapt PR checks to safer checkout defaults
(#65833)
add 6f4134cbe42 [fix](test) stabilize the flaky shuffle_left_join
regression test (#65769)
add 7216a090d8d [fix](topn) Resolve topn lazy materialization column
indexes for aliases (#65759)
add 363c317fc3b [chore](github)update collaborators (#65830)
add 25bdeb83e76 [fix](iceberg) Honor disabled write metrics (#65782)
add 6875132c3d2 [improvement](be) Optimize bit unpacking with PDEP (#65738)
add 1696ff77281 [doc](connector) add project tracking system for catalog
SPI migration
add d0ae3cb7d3d [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add 63989d46680 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add ae92c0e7fda [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add e9d3779c0ef [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 623c808a071 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add ae6ccc6992a update P5 handoff and fix compile issue
add 0cf79390f00 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 2706c7eb2d6 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add 207df0b5aab [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 09bbf92b7c4 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add d92bff215f5 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 1738555ca07 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add a7adebbb593 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add 917874e933b [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add c5c3abf0462 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add c7fb8b41d1f [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 76798cdfe87 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add c3208830f2f [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 233d09feef8 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 631ebfc004d [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 3c290dafbd3 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add ad652613226 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 817bd0f1299 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 28f64a6d3d9 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 6e44c045b7e [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add d14889d4c05 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 2a621c7b1f2 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 3044e390313 [refactor](catalog) Catalog spi 11 hive (#65473)
add f73e549e186 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 0662262e039 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add 5f2bef56b1e [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add 4aa2a58d010 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 89ae658f8f0 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add f730cdb925c [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add ef1af161ac2 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add a288c4ff75d [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add 017dc7ac6f7 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add d63ac1539ba [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add e7331bc449d [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add b9555445177 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add a2a6f696bca [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (568c4bb4571)
\
N -- N -- N refs/heads/branch-catalog-spi (a2a6f696bca)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 3 -
.github/workflows/clang-format.yml | 12 +-
.github/workflows/license-eyes.yml | 22 +--
be/benchmark/benchmark_main.cpp | 4 +
be/benchmark/benchmark_pdep_unpack.hpp | 207 +++++++++++++++++++++
be/src/common/config.cpp | 4 +
be/src/common/config.h | 1 +
be/src/exec/sink/viceberg_merge_sink.cpp | 3 +
.../writer/iceberg/viceberg_partition_writer.cpp | 10 +-
.../writer/iceberg/viceberg_partition_writer.h | 3 +
be/src/format/transformer/vorc_transformer.cpp | 11 +-
be/src/service/http/action/stream_load.cpp | 8 +
be/src/util/bit_packing.inline.h | 22 ++-
be/src/util/pdep_unpack.h | 169 +++++++++++++++++
.../iceberg/iceberg_partition_writer_test.cpp | 107 +++++++++++
.../format/transformer/vorc_transformer_test.cpp | 107 +++++++++++
be/test/util/bit_packing_test.cpp | 130 +++++++++++++
.../iceberg/IcebergWritePlanProvider.java | 11 +-
.../connector/iceberg/IcebergWriterHelper.java | 152 ++++++++++++++-
.../connector/iceberg/IcebergWriterHelperTest.java | 166 ++++++++++++++++-
.../doris/catalog/BuiltinScalarFunctions.java | 5 +-
.../org/apache/doris/httpv2/rest/LoadAction.java | 7 +-
.../org/apache/doris/load/StreamLoadHandler.java | 12 +-
.../functions/scalar/JsonExtractNoQuotes.java | 74 --------
.../expressions/visitor/ScalarFunctionVisitor.java | 5 -
.../plans/physical/PhysicalLazyMaterialize.java | 2 +-
.../doris/statistics/util/StatisticsUtil.java | 14 +-
.../apache/doris/httpv2/rest/LoadActionTest.java | 31 +++
.../apache/doris/load/StreamLoadHandlerTest.java | 102 ++++++++++
.../postprocess/TopnLazyMaterializeTest.java | 25 +++
.../doris/statistics/util/StatisticsUtilTest.java | 54 ++++++
gensrc/thrift/DataSinks.thrift | 4 +
.../multi_cluster/stream_load/stream_load.out | 5 +
.../sql-functions/doc_json_functions_test.out | 2 +-
.../json_functions/test_json_function.out | 11 +-
.../multi_cluster/stream_load/stream_load.groovy | 103 ++++++++++
.../hive/test_hive_topn_lazy_mat.groovy | 9 +
.../kerberos/test_non_catalog_kerberos.groovy | 6 +-
.../distribute/shuffle_left_join.groovy | 18 +-
.../json_functions/test_json_function.groovy | 9 +-
40 files changed, 1501 insertions(+), 149 deletions(-)
create mode 100644 be/benchmark/benchmark_pdep_unpack.hpp
create mode 100644 be/src/util/pdep_unpack.h
create mode 100644
be/test/exec/sink/writer/iceberg/iceberg_partition_writer_test.cpp
create mode 100644 be/test/format/transformer/vorc_transformer_test.cpp
create mode 100644 be/test/util/bit_packing_test.cpp
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/JsonExtractNoQuotes.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
