This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard aaab68ef474 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
omit b505de3d725 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
omit 019e7ae6954 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit 5c744a5898c [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit 0b4f72582e7 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit 67dd69637e3 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit f8f5f3711f9 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit fa29dd5aef0 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit ddacf2a5578 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit 0b3c3c2cb2a [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit 1928d21af13 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 521dd776fee [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit 38dec3d48de [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit e796f8be24f [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit fb2c7becd1e [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit 6e521aa64b2 [refactor](catalog) Catalog spi 11 hive (#65473)
omit fdacbdc9477 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit 4816b0e6771 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit 0fcd85f6a38 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 51c91a90a35 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 69f5ce05491 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit c69921195cd [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 7dc3f4aa35e [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit 590c8deec33 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit 2544b1bb9e2 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 577a19a0ef1 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 337a735c184 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 499e123e156 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit e474c9c7f3a [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit cfd7c00addf [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit 05b1e249fb8 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit 9c0eb9857b7 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 30b21e8fa3d [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 0454fc73a7f [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 8b3c9b5ba7d [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit c1bff95f7fc [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 98dfd62fe28 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit ecfa5bb5bfb update P5 handoff and fix compile issue
omit 0aa26cf6538 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit 9cff003555b [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit 857fd021316 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit f69a97b57bb [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit e498bbacc90 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit bf55619038c [doc](connector) add project tracking system for catalog
SPI migration
add aa559d48dc9 [fix](workflow) Fix BE UT macOS job failing with wrong JDK
version (#65796)
add 88763c11858 [fix](fe) Remove oidc plugin assertions from
AuthenticationPluginManagerTest (#63323)
add e904fbe7d9c [fix](be) Fix array_difference out-of-bounds read on
single-element array (#65855)
add a9816faa665 [refactor](cloud) Rename cloud compute group metadata
class (#65817)
add ab043a433ce [fix](fe) Normalize MySQL JDBC URLs for streaming jobs
(#65647)
add f45e0931636 [fix](runtime-filter) Preserve query semantics during
partition pruning (#65857)
add a8b1fd95e2a [improvement](file) Build native Parquet scan path for
FileScannerV2 (#65674)
add cbf851ddb91 [fix](cloud) Preserve resource IDs when recycling empty
rowsets (#65862)
add b8983bb19dc [doc](connector) add project tracking system for catalog
SPI migration
add 5ca1d15d6c6 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add e0f0c79fa3a [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 3be0bcf2ed1 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add 47bfc11d554 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 046cb643379 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 6bab6000122 update P5 handoff and fix compile issue
add c68e5e47b86 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 72d4c47dc58 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add b47e2ed24a7 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 1001d755516 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 5b7fe17dd6f [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 665edee8c83 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 890014ad706 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add c5d997ee8cc [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add 504397ad88a [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 1d20d3aba20 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 3d946f09fb3 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 6974f595d81 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add e9e99eaae05 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 3ed8be6edc2 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 1e16c5f6f1c [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add 6e8608658ef [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 239d6d441cb [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 10bba885d96 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add a0214b93a54 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 16965f7cda7 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 16013fe2a63 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add ff643d4bc4d [refactor](catalog) Catalog spi 11 hive (#65473)
add 287cfd18da0 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 9df89f774cb [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add e2818c76e5b [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add 40c95c079d5 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 0c97e207c89 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add ed6f1caecf0 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add a1dbf68c9af [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 6608e977e0f [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add fa2da3682ac [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add 91130e6daf7 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add 4e0006f8842 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add d5abf3d44f4 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 9a093765115 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add 76485a6362d [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add 876bbbd5bcc [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add ca840c9dbea [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add d56c8f356c3 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (aaab68ef474)
\
N -- N -- N refs/heads/branch-catalog-spi (d56c8f356c3)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/be-ut-mac.yml | 48 +-
be/src/common/config.cpp | 2 +-
be/src/core/column/column_string.h | 80 +
.../data_type_serde/data_type_datetimev2_serde.cpp | 185 +-
.../data_type_serde/data_type_datetimev2_serde.h | 5 +
.../data_type_serde/data_type_datev2_serde.cpp | 103 +-
.../core/data_type_serde/data_type_datev2_serde.h | 5 +
.../data_type_serde/data_type_decimal_serde.cpp | 265 +-
.../core/data_type_serde/data_type_decimal_serde.h | 5 +
.../data_type_serde/data_type_number_serde.cpp | 339 +-
.../core/data_type_serde/data_type_number_serde.h | 5 +
be/src/core/data_type_serde/data_type_serde.cpp | 13 +
be/src/core/data_type_serde/data_type_serde.h | 15 +
.../data_type_serde/data_type_string_serde.cpp | 114 +
.../core/data_type_serde/data_type_string_serde.h | 5 +
.../core/data_type_serde/data_type_time_serde.cpp | 107 +
be/src/core/data_type_serde/data_type_time_serde.h | 5 +
.../data_type_timestamptz_serde.cpp | 156 +-
.../data_type_serde/data_type_timestamptz_serde.h | 5 +
.../data_type_serde/data_type_varbinary_serde.cpp | 69 +
.../data_type_serde/data_type_varbinary_serde.h | 6 +
.../core/data_type_serde/parquet_decode_source.h | 392 ++
be/src/core/data_type_serde/parquet_timestamp.h | 95 +
be/src/exec/scan/file_scanner_v2.cpp | 103 +-
be/src/exec/scan/file_scanner_v2.h | 16 +
be/src/exec/scan/scanner.h | 2 +-
.../function/array/function_array_difference.h | 13 +-
be/src/exprs/vcompound_pred.h | 8 +
be/src/exprs/vectorized_fn_call.cpp | 151 +-
be/src/exprs/vectorized_fn_call.h | 5 +
be/src/exprs/vexpr.h | 13 +
be/src/format_v2/AGENTS.md | 110 +
be/src/format_v2/column_mapper.cpp | 37 +-
.../delimited_text/delimited_text_reader.cpp | 21 +-
.../delimited_text/delimited_text_reader.h | 1 +
.../format_v2/expr/equality_delete_predicate.cpp | 61 +-
be/src/format_v2/file_reader.cpp | 4 +
be/src/format_v2/file_reader.h | 8 +
be/src/format_v2/jni/jdbc_reader.cpp | 19 +-
be/src/format_v2/jni/jni_table_reader.cpp | 48 +-
be/src/format_v2/jni/jni_table_reader.h | 2 +
.../format_v2/jni/trino_connector_jni_reader.cpp | 11 +-
be/src/format_v2/json/json_reader.cpp | 37 +-
be/src/format_v2/json/json_reader.h | 8 +
be/src/format_v2/native/native_reader.cpp | 40 +-
be/src/format_v2/native/native_reader.h | 6 +
be/src/format_v2/orc/orc_file_input_stream.cpp | 37 +-
be/src/format_v2/orc/orc_reader.cpp | 44 +-
be/src/format_v2/orc/orc_reader.h | 1 +
be/src/format_v2/parquet/native_schema_desc.cpp | 870 +++++
be/src/format_v2/parquet/native_schema_desc.h | 183 +
be/src/format_v2/parquet/native_schema_node.cpp | 127 +
be/src/format_v2/parquet/native_schema_node.h | 93 +
be/src/format_v2/parquet/parquet_column_schema.cpp | 610 +---
be/src/format_v2/parquet/parquet_column_schema.h | 19 +-
be/src/format_v2/parquet/parquet_file_context.cpp | 976 +++--
be/src/format_v2/parquet/parquet_file_context.h | 166 +-
be/src/format_v2/parquet/parquet_profile.cpp | 124 +-
be/src/format_v2/parquet/parquet_profile.h | 75 +-
be/src/format_v2/parquet/parquet_reader.cpp | 282 +-
be/src/format_v2/parquet/parquet_reader.h | 3 +-
be/src/format_v2/parquet/parquet_scan.cpp | 1343 +++++--
be/src/format_v2/parquet/parquet_scan.h | 135 +-
be/src/format_v2/parquet/parquet_statistics.cpp | 1684 ++++-----
be/src/format_v2/parquet/parquet_statistics.h | 105 +-
be/src/format_v2/parquet/parquet_type.cpp | 325 +-
be/src/format_v2/parquet/parquet_type.h | 18 +-
be/src/format_v2/parquet/reader/column_reader.cpp | 605 +---
be/src/format_v2/parquet/reader/column_reader.h | 207 +-
.../parquet/reader/count_column_reader.cpp | 225 ++
.../format_v2/parquet/reader/count_column_reader.h | 72 +
.../parquet/reader/list_column_reader.cpp | 230 --
.../format_v2/parquet/reader/list_column_reader.h | 57 -
.../format_v2/parquet/reader/map_column_reader.cpp | 285 --
.../format_v2/parquet/reader/map_column_reader.h | 61 -
.../reader/native/block_split_bloom_filter.cpp | 116 +
.../reader/native/block_split_bloom_filter.h | 54 +
.../parquet/reader/native/bool_plain_decoder.cpp | 99 +
.../parquet/reader/native/bool_plain_decoder.h | 107 +
.../parquet/reader/native/bool_rle_decoder.cpp | 94 +
.../parquet/reader/native/bool_rle_decoder.h | 61 +
.../reader/native/byte_array_dict_decoder.cpp | 67 +
.../reader/native/byte_array_dict_decoder.h | 45 +
.../reader/native/byte_array_plain_decoder.cpp | 131 +
.../reader/native/byte_array_plain_decoder.h | 73 +
.../reader/native/byte_stream_split_decoder.cpp | 82 +
.../reader/native/byte_stream_split_decoder.h | 60 +
.../parquet/reader/native/column_chunk_reader.cpp | 1785 ++++++++++
.../parquet/reader/native/column_chunk_reader.h | 367 ++
.../parquet/reader/native/column_reader.cpp | 1887 ++++++++++
.../parquet/reader/native/column_reader.h | 654 ++++
be/src/format_v2/parquet/reader/native/common.cpp | 196 +
be/src/format_v2/parquet/reader/native/common.h | 111 +
be/src/format_v2/parquet/reader/native/decoder.cpp | 156 +
be/src/format_v2/parquet/reader/native/decoder.h | 345 ++
.../reader/native/delta_bit_pack_decoder.cpp | 180 +
.../parquet/reader/native/delta_bit_pack_decoder.h | 812 +++++
.../reader/native/fix_length_dict_decoder.hpp | 64 +
.../reader/native/fix_length_plain_decoder.cpp | 93 +
.../reader/native/fix_length_plain_decoder.h | 51 +
.../parquet/reader/native/level_decoder.cpp | 271 ++
.../parquet/reader/native/level_decoder.h | 87 +
.../parquet/reader/native/level_reader.cpp | 222 ++
.../format_v2/parquet/reader/native/level_reader.h | 72 +
.../parquet/reader/native/page_reader.cpp | 332 ++
.../format_v2/parquet/reader/native/page_reader.h | 303 ++
.../parquet/reader/native_column_reader.cpp | 762 ++++
.../parquet/reader/native_column_reader.h | 150 +
.../parquet/reader/nested_column_materializer.cpp | 70 -
.../parquet/reader/nested_column_materializer.h | 45 -
.../parquet/reader/parquet_leaf_reader.cpp | 803 -----
.../format_v2/parquet/reader/parquet_leaf_reader.h | 173 -
.../parquet/reader/row_position_column_reader.cpp | 1 +
.../parquet/reader/scalar_column_reader.cpp | 584 ---
.../parquet/reader/scalar_column_reader.h | 110 -
.../parquet/reader/struct_column_reader.cpp | 287 --
.../parquet/reader/struct_column_reader.h | 65 -
be/src/format_v2/parquet/selection_vector.h | 58 +-
be/src/format_v2/table/hive_reader.cpp | 15 +-
be/src/format_v2/table/hudi_reader.cpp | 35 +-
be/src/format_v2/table/hudi_reader.h | 11 +
.../iceberg_position_delete_sys_table_reader.cpp | 16 +-
.../iceberg_position_delete_sys_table_reader.h | 1 +
be/src/format_v2/table/iceberg_reader.cpp | 46 +-
be/src/format_v2/table/paimon_reader.cpp | 33 +-
be/src/format_v2/table/paimon_reader.h | 11 +
be/src/format_v2/table/remote_doris_reader.cpp | 51 +-
be/src/format_v2/table/remote_doris_reader.h | 7 +
be/src/format_v2/table_reader.cpp | 116 +-
be/src/format_v2/table_reader.h | 172 +-
be/src/io/cache/block_file_cache_profile.cpp | 23 +-
be/src/io/cache/block_file_cache_profile.h | 5 +-
be/src/io/fs/buffered_reader.cpp | 11 +-
be/src/io/fs/buffered_reader.h | 8 +-
be/src/io/fs/hdfs_file_reader.cpp | 6 +-
be/src/io/fs/hdfs_file_reader.h | 1 +
be/src/io/fs/s3_file_reader.cpp | 6 +-
be/src/runtime/file_scan_profile.h | 55 +
be/src/util/block_compression.cpp | 12 +-
be/src/util/cpu_info.cpp | 11 +
be/src/util/cpu_info.h | 4 +
.../cloud/cloud_tablet_query_prefer_cache_test.cpp | 13 +-
.../cloud_tablet_query_with_tolerance_test.cpp | 25 +-
be/test/cloud/cloud_tablet_test.cpp | 1 -
.../data_type_serde_decoded_values_test.cpp | 49 +-
.../data_type_serde_parquet_test.cpp | 891 +++++
be/test/core/jsonb/jsonb_document_cast_test.cpp | 6 +-
be/test/exec/column_type_convert_test.cpp | 6 +-
be/test/exec/scan/file_scanner_v2_test.cpp | 21 +
be/test/exec/scan/scanner_context_test.cpp | 4 +-
.../function/cast/cast_to_string_api_test.cpp | 8 +-
be/test/exprs/vexpr_test.cpp | 4 +-
be/test/format_v2/column_mapper_test.cpp | 32 +
.../format_v2/delimited_text/csv_reader_test.cpp | 16 +-
.../format_v2/delimited_text/text_reader_test.cpp | 44 +-
be/test/format_v2/jni/jni_table_reader_test.cpp | 43 +-
.../format_v2/orc/orc_file_input_stream_test.cpp | 47 +-
be/test/format_v2/orc/orc_reader_test.cpp | 144 +-
be/test/format_v2/parquet/native_decoder_test.cpp | 3742 ++++++++++++++++++++
.../parquet/parquet_column_reader_test.cpp | 3682 -------------------
.../format_v2/parquet/parquet_leaf_reader_test.cpp | 506 ---
.../parquet/parquet_page_cache_range_test.cpp | 141 +-
.../parquet/parquet_reader_control_test.cpp | 1146 +-----
be/test/format_v2/parquet/parquet_reader_test.cpp | 1238 +++++--
be/test/format_v2/parquet/parquet_scan_test.cpp | 986 ++++--
be/test/format_v2/parquet/parquet_schema_test.cpp | 1010 +++---
.../parquet/parquet_serde_reader_test.cpp | 459 ---
.../format_v2/parquet/parquet_statistics_test.cpp | 1445 +++-----
be/test/format_v2/parquet/parquet_type_test.cpp | 491 +--
be/test/format_v2/table/hudi_reader_test.cpp | 61 +
...eberg_position_delete_sys_table_reader_test.cpp | 63 +
be/test/format_v2/table/iceberg_reader_test.cpp | 29 +-
be/test/format_v2/table/paimon_reader_test.cpp | 62 +
be/test/format_v2/table_reader_test.cpp | 133 +
.../block_file_cache_profile_reporter_test.cpp | 24 +
be/test/io/cache/block_file_cache_test.cpp | 8 +-
be/test/io/cache/block_file_cache_test_common.h | 1 -
be/test/io/cache/lru_queue_test.cpp | 1 -
.../runtime_profile_counter_tree_node_test.cpp | 4 +-
be/test/runtime/runtime_profile_test.cpp | 10 +-
.../compaction/compaction_permit_limiter_test.cpp | 2 +
.../storage/index/ann/ann_index_edge_case_test.cpp | 8 +-
.../storage/segment/inverted_index_reader_test.cpp | 14 +-
be/test/udf/python/python_env_test.cpp | 4 +-
cloud/src/recycler/recycler.cpp | 17 +-
cloud/test/recycler_test.cpp | 38 +
docs/file-scanner-v2-code-review-guide.md | 150 +-
docs/file-scanner-v2-design.md | 17 +
docs/file-scanner-v2-parquet-scan-design.md | 445 ++-
.../trino-connector-scanner/pom.xml | 7 +
fe/fe-common/pom.xml | 11 +-
fe/fe-connector/fe-connector-trino/pom.xml | 5 +-
fe/fe-core/pom.xml | 17 +-
.../doris/cloud/catalog/BalanceTypeEnum.java | 3 +-
...omputeGroup.java => CloudComputeGroupMeta.java} | 28 +-
.../cloud/catalog/CloudInstanceStatusChecker.java | 30 +-
.../doris/cloud/catalog/CloudTabletRebalancer.java | 10 +-
.../doris/cloud/system/CloudSystemInfoService.java | 43 +-
.../insert/streaming/StreamingInsertJob.java | 17 +-
.../streaming/StreamingJdbcUrlNormalizer.java | 83 +
.../offset/jdbc/JdbcTvfSourceOffsetProvider.java | 3 +
.../RuntimeFilterPartitionPruneClassifier.java | 4 +
.../rules/implementation/AggregateStrategies.java | 24 +-
.../plans/commands/AlterComputeGroupCommand.java | 4 +-
.../trees/plans/commands/ShowClustersCommand.java | 8 +-
.../trees/plans/commands/WarmUpClusterCommand.java | 6 +-
.../CdcStreamTableValuedFunction.java | 10 +-
.../cloud/WarmUpClusterOnTablesParseTest.java | 20 +-
...oupTest.java => CloudComputeGroupMetaTest.java} | 135 +-
.../catalog/CloudInstanceStatusCheckerTest.java | 10 +-
.../cloud/system/CloudSystemInfoServiceTest.java | 104 +-
.../datasource/scan/FileQueryScanNodeTest.java | 1 +
.../streaming/StreamingJdbcUrlNormalizerTest.java | 78 +
.../jdbc/JdbcTvfSourceOffsetProviderTest.java | 46 +
.../doris/mysql/privilege/CloudAuthTest.java | 8 +-
.../RuntimeFilterPartitionPruneClassifierTest.java | 37 +-
.../rewrite/PhysicalStorageLayerAggregateTest.java | 84 +
.../commands/AlterComputeGroupCommandTest.java | 8 +-
.../CdcStreamTableValuedFunctionTest.java | 15 +
fe/fe-trino-connector-common/pom.xml | 61 +
.../doris/trinoconnector/TrinoColumnMetadata.java | 0
.../TrinoConnectorPluginManager.java | 0
.../TrinoConnectorServicesProvider.java | 0
fe/pom.xml | 6 +
.../source/reader/mysql/MySqlSourceReader.java | 8 +
.../source/reader/mysql/MySqlYearConverter.java | 70 +
.../source/reader/mysql/MySqlSourceReaderTest.java | 29 +
.../reader/mysql/MySqlYearConverterTest.java | 75 +
.../tvf/test_hdfs_parquet_group6.out | 16 +-
.../cdc/test_streaming_mysql_job_all_type.out | 9 +-
.../test_streaming_mysql_job_integer_boundary.out | 33 +-
.../cdc/tvf/test_cdc_stream_tvf_mysql.out | 13 +-
...ing_job_cdc_stream_mysql_jdbc_type_defaults.out | 14 +
.../runtime_filter/rf_partition_pruning.out | 6 +
.../test_array_functions_of_array_difference.out | 12 +
.../hive/test_orc_lazy_mat_profile.groovy | 32 +-
.../cdc/test_streaming_mysql_job_all_type.groovy | 4 +-
..._streaming_mysql_job_charset_and_strings.groovy | 6 +-
.../test_streaming_mysql_job_create_alter.groovy | 37 +-
...est_streaming_mysql_job_integer_boundary.groovy | 41 +-
.../test_streaming_mysql_job_special_offset.groovy | 11 +-
.../cdc/tvf/test_cdc_stream_tvf_mysql.groovy | 11 +-
..._job_cdc_stream_mysql_jdbc_type_defaults.groovy | 126 +
.../runtime_filter/rf_partition_pruning.groovy | 53 +
...test_array_functions_of_array_difference.groovy | 6 +
245 files changed, 27866 insertions(+), 15414 deletions(-)
create mode 100644 be/src/core/data_type_serde/parquet_decode_source.h
create mode 100644 be/src/core/data_type_serde/parquet_timestamp.h
create mode 100644 be/src/format_v2/parquet/native_schema_desc.cpp
create mode 100644 be/src/format_v2/parquet/native_schema_desc.h
create mode 100644 be/src/format_v2/parquet/native_schema_node.cpp
create mode 100644 be/src/format_v2/parquet/native_schema_node.h
create mode 100644 be/src/format_v2/parquet/reader/count_column_reader.cpp
create mode 100644 be/src/format_v2/parquet/reader/count_column_reader.h
delete mode 100644 be/src/format_v2/parquet/reader/list_column_reader.cpp
delete mode 100644 be/src/format_v2/parquet/reader/list_column_reader.h
delete mode 100644 be/src/format_v2/parquet/reader/map_column_reader.cpp
delete mode 100644 be/src/format_v2/parquet/reader/map_column_reader.h
create mode 100644
be/src/format_v2/parquet/reader/native/block_split_bloom_filter.cpp
create mode 100644
be/src/format_v2/parquet/reader/native/block_split_bloom_filter.h
create mode 100644
be/src/format_v2/parquet/reader/native/bool_plain_decoder.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/bool_plain_decoder.h
create mode 100644 be/src/format_v2/parquet/reader/native/bool_rle_decoder.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/bool_rle_decoder.h
create mode 100644
be/src/format_v2/parquet/reader/native/byte_array_dict_decoder.cpp
create mode 100644
be/src/format_v2/parquet/reader/native/byte_array_dict_decoder.h
create mode 100644
be/src/format_v2/parquet/reader/native/byte_array_plain_decoder.cpp
create mode 100644
be/src/format_v2/parquet/reader/native/byte_array_plain_decoder.h
create mode 100644
be/src/format_v2/parquet/reader/native/byte_stream_split_decoder.cpp
create mode 100644
be/src/format_v2/parquet/reader/native/byte_stream_split_decoder.h
create mode 100644
be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/column_chunk_reader.h
create mode 100644 be/src/format_v2/parquet/reader/native/column_reader.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/column_reader.h
create mode 100644 be/src/format_v2/parquet/reader/native/common.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/common.h
create mode 100644 be/src/format_v2/parquet/reader/native/decoder.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/decoder.h
create mode 100644
be/src/format_v2/parquet/reader/native/delta_bit_pack_decoder.cpp
create mode 100644
be/src/format_v2/parquet/reader/native/delta_bit_pack_decoder.h
create mode 100644
be/src/format_v2/parquet/reader/native/fix_length_dict_decoder.hpp
create mode 100644
be/src/format_v2/parquet/reader/native/fix_length_plain_decoder.cpp
create mode 100644
be/src/format_v2/parquet/reader/native/fix_length_plain_decoder.h
create mode 100644 be/src/format_v2/parquet/reader/native/level_decoder.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/level_decoder.h
create mode 100644 be/src/format_v2/parquet/reader/native/level_reader.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/level_reader.h
create mode 100644 be/src/format_v2/parquet/reader/native/page_reader.cpp
create mode 100644 be/src/format_v2/parquet/reader/native/page_reader.h
create mode 100644 be/src/format_v2/parquet/reader/native_column_reader.cpp
create mode 100644 be/src/format_v2/parquet/reader/native_column_reader.h
delete mode 100644
be/src/format_v2/parquet/reader/nested_column_materializer.cpp
delete mode 100644 be/src/format_v2/parquet/reader/nested_column_materializer.h
delete mode 100644 be/src/format_v2/parquet/reader/parquet_leaf_reader.cpp
delete mode 100644 be/src/format_v2/parquet/reader/parquet_leaf_reader.h
delete mode 100644 be/src/format_v2/parquet/reader/scalar_column_reader.cpp
delete mode 100644 be/src/format_v2/parquet/reader/scalar_column_reader.h
delete mode 100644 be/src/format_v2/parquet/reader/struct_column_reader.cpp
delete mode 100644 be/src/format_v2/parquet/reader/struct_column_reader.h
create mode 100644 be/src/runtime/file_scan_profile.h
create mode 100644
be/test/core/data_type_serde/data_type_serde_parquet_test.cpp
create mode 100644 be/test/format_v2/parquet/native_decoder_test.cpp
delete mode 100644 be/test/format_v2/parquet/parquet_column_reader_test.cpp
delete mode 100644 be/test/format_v2/parquet/parquet_leaf_reader_test.cpp
delete mode 100644 be/test/format_v2/parquet/parquet_serde_reader_test.cpp
create mode 100644
be/test/format_v2/table/iceberg_position_delete_sys_table_reader_test.cpp
rename
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/{ComputeGroup.java =>
CloudComputeGroupMeta.java} (89%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingJdbcUrlNormalizer.java
rename
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/{ComputeGroupTest.java
=> CloudComputeGroupMetaTest.java} (61%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/StreamingJdbcUrlNormalizerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/job/offset/jdbc/JdbcTvfSourceOffsetProviderTest.java
create mode 100644 fe/fe-trino-connector-common/pom.xml
rename fe/{fe-common =>
fe-trino-connector-common}/src/main/java/org/apache/doris/trinoconnector/TrinoColumnMetadata.java
(100%)
rename fe/{fe-common =>
fe-trino-connector-common}/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorPluginManager.java
(100%)
rename fe/{fe-common =>
fe-trino-connector-common}/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorServicesProvider.java
(100%)
create mode 100644
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlYearConverter.java
create mode 100644
fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlYearConverterTest.java
create mode 100644
regression-test/data/job_p0/streaming_job/cdc/tvf/test_streaming_job_cdc_stream_mysql_jdbc_type_defaults.out
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/tvf/test_streaming_job_cdc_stream_mysql_jdbc_type_defaults.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
