This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard abfb7058f90 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64256)
omit 26381c11a81 [feat](connector) P4 maxcompute connector full adoption +
live cutover via catalog SPI (T01-T06) (#64253)
omit 486f1319005 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit 591801208d7 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 05ee43b16bc [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit aa97b737180 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit 78d1cf787d9 [doc](connector) add project tracking system for catalog
SPI migration
add 2366edffcc6 [bug](iceberg) fix can't get migrated Iceberg tables
format type (#64134)
add 3ece7b5ddc6 [Enhancement](pyudf) Enhance Python process poll init and
repair (#64039)
add 952aede1883 [fix](be) Rename CPU time profile counter (#64238)
add fa91940f8b7 [fix](cloud) Validate recycle rowset key state during
commit rowset (#63985)
add 5ff34855dd2 [fix](function) keep struct_element as an alias of
element_at (#64261)
add 1dd430be7f6 [fix](variant) Reject COUNT DISTINCT on variant arguments
(#63479)
add 6fc8f333400 [fix](fe) Improve MaxCompute catalog validation (#64119)
new a3988c72cfe [doc](connector) add project tracking system for catalog
SPI migration
new 58c0138610f [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
new 19451cb41a0 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
new 826114f7492 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
new 035bbdeee1a [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
new 8b7c73cfd20 [feat](connector) P4 maxcompute connector full adoption +
live cutover via catalog SPI (T01-T06) (#64253)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (abfb7058f90)
\
N -- N -- N refs/heads/branch-catalog-spi (8b7c73cfd20)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/exchange/vdata_stream_recvr.cpp | 2 +-
.../function/array/function_array_element.cpp | 2 +
be/src/udf/python/python_server.cpp | 570 +++++++++++----
be/src/udf/python/python_server.h | 50 +-
be/src/udf/python/python_udf_runtime.cpp | 220 +++++-
be/src/udf/python/python_udf_runtime.h | 15 +-
be/test/exec/pipeline/vdata_stream_recvr_test.cpp | 18 +
be/test/runtime/runtime_profile_test.cpp | 20 +
be/test/udf/python/python_server_test.cpp | 364 ++++++++-
be/test/udf/python/python_udf_runtime_test.cpp | 89 +++
cloud/src/meta-service/meta_service.cpp | 125 ++--
cloud/test/meta_service_test.cpp | 87 ++-
.../create_preinstalled_scripts/iceberg/run10.sql | 4 +-
.../doris/maxcompute/MaxComputeJniScanner.java | 1 +
.../doris/maxcompute/MaxComputeJniWriter.java | 1 +
fe/be-java-extensions/preload-extensions/pom.xml | 12 -
fe/fe-common/pom.xml | 24 +-
.../apache/doris/common}/maxcompute/MCUtils.java | 4 +-
.../maxcompute/MaxComputeDorisConnector.java | 67 +-
.../maxcompute/MaxComputeScanPlanProvider.java | 4 -
.../maxcompute/MaxComputeTableHandle.java | 24 -
.../maxcompute/MaxComputeWritePlanProvider.java | 3 -
.../MaxComputeConnectorProviderTest.java | 132 ----
.../MaxComputeConnectorTransactionTest.java | 36 -
.../maxcompute/MaxComputeScanPlanProviderTest.java | 36 -
fe/fe-core/pom.xml | 27 +-
.../doris/catalog/BuiltinScalarFunctions.java | 3 +-
.../apache/doris/datasource/ExternalCatalog.java | 3 +
.../doris/datasource/ExternalMetaCacheMgr.java | 8 +
.../doris/datasource/iceberg/IcebergUtils.java | 40 +-
.../doris/datasource/maxcompute/MCTransaction.java | 266 +++++++
.../maxcompute/MaxComputeExternalCatalog.java | 524 +++++++++++++
.../MaxComputeExternalDatabase.java} | 27 +-
.../maxcompute/MaxComputeExternalMetaCache.java | 115 +++
.../maxcompute/MaxComputeExternalTable.java | 347 +++++++++
.../maxcompute/MaxComputeMetadataOps.java | 565 ++++++++++++++
.../maxcompute/MaxComputeSchemaCacheValue.java | 67 ++
.../datasource/maxcompute/McStructureHelper.java | 298 ++++++++
.../maxcompute/source/MaxComputeScanNode.java | 814 +++++++++++++++++++++
.../source/MaxComputeSplit.java} | 29 +-
.../metacache/ExternalMetaCacheRouteResolver.java | 6 +
...leSink.java => UnboundMaxComputeTableSink.java} | 77 +-
.../nereids/analyzer/UnboundTableSinkCreator.java | 9 +
.../glue/translator/PhysicalPlanTranslator.java | 18 +
.../nereids/processor/post/ShuffleKeyPruner.java | 15 +
.../pre/TurnOffPageCacheForInsertIntoSelect.java | 8 +
.../nereids/properties/RequestPropertyDeriver.java | 12 +
.../org/apache/doris/nereids/rules/RuleSet.java | 3 +
.../doris/nereids/rules/analysis/BindSink.java | 65 ++
.../rules/expression/ExpressionRewrite.java | 9 +
...uteTableSinkToPhysicalMaxComputeTableSink.java} | 16 +-
.../trees/expressions/functions/agg/Count.java | 30 +-
.../functions/agg/MultiDistinctCount.java | 7 +
.../plans/commands/ShowPartitionsCommand.java | 24 +
.../trees/plans/commands/info/CreateTableInfo.java | 5 +
.../commands/insert/InsertIntoTableCommand.java | 34 +
.../insert/InsertOverwriteTableCommand.java | 24 +
.../trees/plans/commands/insert/InsertUtils.java | 5 +
.../commands/insert/MCInsertCommandContext.java | 84 +++
.../plans/commands/insert/MCInsertExecutor.java | 84 +++
...leSink.java => LogicalMaxComputeTableSink.java} | 59 +-
...eSink.java => PhysicalMaxComputeTableSink.java} | 103 +--
.../nereids/trees/plans/visitor/SinkVisitor.java | 15 +
.../apache/doris/planner/MaxComputeTableSink.java | 113 +++
.../doris/tablefunction/MetadataGenerator.java | 18 +
.../PartitionValuesTableValuedFunction.java | 4 +-
.../PartitionsTableValuedFunction.java | 9 +
...ctionManager.java => MCTransactionManager.java} | 16 +-
.../transaction/TransactionManagerFactory.java | 5 +
.../ExternalMetaCacheRouteResolverTest.java | 6 +
.../datasource/maxcompute/MCTransactionTest.java | 54 ++
.../maxcompute/MaxComputeExternalCatalogTest.java | 146 ++++
.../MaxComputeExternalMetaCacheTest.java | 139 ++++
.../maxcompute/source/MaxComputeScanNodeTest.java | 463 ++++++++++++
.../trees/expressions/functions/agg/CountTest.java | 84 +++
.../doris/service/FrontendServiceImplTest.java | 11 +-
.../transaction/CommitDataSerializerTest.java | 21 +
fe/pom.xml | 10 +-
plan-doc/HANDOFF.md | 99 ---
plan-doc/PROGRESS.md | 8 +-
plan-doc/deviations-log.md | 4 +-
plan-doc/task-list-batchD-redline-gaps.md | 2 +-
.../designs/P4-batchD-maxcompute-removal-design.md | 73 +-
.../struct_functions/test_struct_functions.out | 32 +
.../struct_functions/test_struct_functions.groovy | 7 +
.../suites/query_profile/scanner_profile.groovy | 9 +-
.../test_variant_count_distinct.groovy} | 29 +-
87 files changed, 6167 insertions(+), 950 deletions(-)
rename
fe/{be-java-extensions/max-compute-connector/src/main/java/org/apache/doris =>
fe-common/src/main/java/org/apache/doris/common}/maxcompute/MCUtils.java (97%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalog.java
copy
fe/fe-core/src/main/java/org/apache/doris/datasource/{lakesoul/LakeSoulExternalDatabase.java
=> maxcompute/MaxComputeExternalDatabase.java} (58%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCache.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeSchemaCacheValue.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/McStructureHelper.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java
copy fe/fe-core/src/main/java/org/apache/doris/datasource/{SplitCreator.java
=> maxcompute/source/MaxComputeSplit.java} (58%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/{UnboundConnectorTableSink.java
=> UnboundMaxComputeTableSink.java} (50%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/{LogicalConnectorTableSinkToPhysicalConnectorTableSink.java
=> LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink.java} (68%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertCommandContext.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/{LogicalIcebergTableSink.java
=> LogicalMaxComputeTableSink.java} (68%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/{PhysicalIcebergTableSink.java
=> PhysicalMaxComputeTableSink.java} (51%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/MaxComputeTableSink.java
copy
fe/fe-core/src/main/java/org/apache/doris/transaction/{IcebergTransactionManager.java
=> MCTransactionManager.java} (64%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MCTransactionTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCacheTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNodeTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/agg/CountTest.java
copy regression-test/suites/{query_p0/limit/OffsetInSubqueryWithJoin.groovy =>
variant_p0/test_variant_count_distinct.groovy} (56%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
