airborne12 opened a new pull request, #70357: URL: https://github.com/apache/doris/pull/70357
### What problem does this PR solve? Issue Number: No separate issue is linked by the source PRs. Related PRs: [#62181](https://github.com/apache/doris/pull/62181), [#69442](https://github.com/apache/doris/pull/69442), [#69443](https://github.com/apache/doris/pull/69443) Backport #62181 with the two follow-up correctness fixes #69442 and #69443. Partition DROP INDEX deletes physical inverted-index files from selected partitions while retaining the table index definition. Mixed ALTER work and unfinished BUILD/DROP jobs on the same index are rejected before metadata changes. The older target needs a DropIndexClause adapter and a minimal early full-table conflict check. ### Release note Support `DROP INDEX ... PARTITION/PARTITIONS (...)` for inverted indexes. Preserve the table index definition and allow a later BUILD INDEX to restore physical indexes. Reject mixed partition ALTER work and concurrent BUILD/DROP jobs on the same index. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. Partition-scoped physical deletion and the release-note behavior above. - Does this need documentation? - [x] Yes. The new partition DROP INDEX syntax needs release documentation; the source #62181 does not link a documentation PR. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label ### Validation - `./build.sh --fe -j8`: passed with JDK 17, including FE production and test compilation. - `./run-fe-ut.sh --run org.apache.doris.alter.SchemaChangeHandlerTest,org.apache.doris.alter.IndexChangeJobTest,org.apache.doris.nereids.parser.NereidsParserTest`: 127 passed, zero failures/errors/skips. All eight source-added FE cases executed, including full-table and partition BUILD/DROP conflict assertions. - `./run-regression-test.sh --conf <isolated-conf> --run -d inverted_index_p0/index_change -s test_drop_index_on_partition -parallel 1 -suiteParallel 1`: passed with zero failed/fatal/skipped suites. All ten original scenarios executed against an isolated local-mode FE/ASAN BE cluster. Four explicit BUILD calls adapt setup to this target's metadata-only CREATE INDEX path; the original assertions remain. - This PR changes FE/parser/test code only. The ASAN backend was reused from successful TeamCity compile `1094591` for [#70325](https://github.com/apache/doris/pull/70325), build head `554526b3f485320686b28d65f850526c5b554000`, whose backend/build dependency source trees equal target base `a8f9f5d419deaa4c1e04db575554d6cfdd46f48a`. The [downloaded artifact](https://opensource-pipeline.oss-cn-hongkong.aliyuncs.com/compile_result/70325_554526b3f485320686b28d65f850526c5b554000.tar.gz) was checked for ASAN via its build log, runtime version and `__asan_init` symbol. The native binary used by the cluster is byte-identical to that artifact. Only this backport's own FE build is used. No backend or protocol change is introduced. - No sanitizer diagnostic was observed during the integration run. No sanitizer suppression changed. Cloud-mode integration was not run; existing FE cloud/local paths are preserved. - Complete PR English check passed with zero data exceptions. Full current-target merged-tree FE reactor Checkstyle passed. Checked head `ab4a348ef31fdd23318b65fc6a911f28ec48b85b`, target base `a8f9f5d419deaa4c1e04db575554d6cfdd46f48a`. <details> <summary>Source range-diff review and per-hunk backport audit</summary> ### Range-diff review Each excerpt comes from `git range-diff --creation-factor=10000 <source>^! <pick>^!`. The hunk audit below covers every original source hunk and the final compatibility changes. #62181: ```diff 1: 791c56d85db ! 1: 558d5832ea8 [feature](inverted index) support DROP INDEX ON PARTITION for inverted index (#62181) - | RENAME COLUMN name=identifier newName=identifier #renameColumnClause + | RENAME COLUMN name=qualifiedName TO? newName=identifier #renameColumnClause - } else if (alterOp instanceof DropIndexOp) { ``` Keep the target parser context and legacy AlterClause dispatch. Pass PartitionNamesInfo through DropIndexClause, retain full-table behavior, and run every original feature case. The final regression compatibility commit explicitly builds physical indexes before the original waits and assertions. #69442: ```diff 1: adecd627ab0 ! 1: ade3f32345c [fix](fe) Reject DROP INDEX ... PARTITION combined with other ALTER clauses (#69442) - // that can reject those jobs before any clause below touches table metadata (processDropColumn - // updates the bloom filter columns in place), so a rejected statement leaves the table unchanged. - checkLightDropIndexPreconditions(db, olapTable, alterOps, enableAddIndexForNewData); ``` Use the legacy DropIndexClause and reject mixed partition work before metadata mutation. Keep all mixed-statement and metadata/job-count assertions; shorten comments. #69443: ```diff 1: 8ebb90998a5 ! 1: a7c700ae87f [fix](fe) Treat an unfinished index change job of the same index as a conflict in either direction (#69443) - } - if (hasIndexChangeJobOnPartition(originIndexId, db.getId(), olapTable.getId(), -- partitionName, alterIndexes, isDropOp)) { ``` Keep the direction-independent predicate and both full-table and partition rejection assertions. The 4.1 target additionally needs an early full-table conflict check because its source pre-check helper is absent. The final compatibility commit also removes the obsolete direction argument from the legacy partition task-submission call. Shorten the test comment. ### Original-source hunk audit Hunk headers are copied verbatim from the original GitHub PR diff. Source #62181: `791c56d85db0648ae8911616a92d138f87ab7b11` → `558d5832ea843761200d1cdf3e41973ca8d4e099` `fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4` |Original source hunk header|Disposition| |---|---| |`@@ -352,7 +352,8 @@ supportedDropStatement`|Ported: Source changed lines are present, including behavior already implemented on the target.| |`@@ -764,7 +765,7 @@ alterTableClause`|Ported: Source changed lines are present, including behavior already implemented on the target.| `fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java` |Original source hunk header|Disposition| |---|---| |`@@ -2349,37 +2349,80 @@ public int getAsInt() {`|Adapted: Use AlterClause/DropIndexClause in the branch-4.1 loop. Add physical-only partition deletion and retain target full-table dispatch, without inheriting the unrelated branch-4.2 pre-check.| |`@@ -2409,7 +2452,7 @@ public int getAsInt() {`|Ported: Source changed lines are present, including behavior already implemented on the target.| `fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java` |Original source hunk header|Disposition| |---|---| |`@@ -6287,14 +6287,26 @@ public Command visitBuildIndex(BuildIndexContext ctx) {`|Ported: Source changed lines are present, including behavior already implemented on the target.| `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropIndexOp.java` |Original source hunk header|Disposition| |---|---| |`@@ -18,6 +18,7 @@`|Adapted: Use org.apache.doris.info.PartitionNamesInfo and pass it through translateToLegacyAlterClause(). Extend target analysis/DropIndexClause with the same constructor, accessors, validation and SQL rendering.| |`@@ -25,6 +26,8 @@`|Ported: Source changed lines are present, including behavior already implemented on the target.| |`@@ -36,12 +39,26 @@ public class DropIndexOp extends AlterTableOp {`|Adapted: Use org.apache.doris.info.PartitionNamesInfo and pass it through translateToLegacyAlterClause(). Extend target analysis/DropIndexClause with the same constructor, accessors, validation and SQL rendering.| |`@@ -60,6 +77,16 @@ public boolean isAlter() {`|Ported: Source changed lines are present, including behavior already implemented on the target.| |`@@ -73,6 +100,18 @@ public void validate(ConnectContext ctx) throws UserException {`|Ported: Source changed lines are present, including behavior already implemented on the target.| |`@@ -92,6 +131,9 @@ public String toSql() {`|Ported: Source changed lines are present, including behavior already implemented on the target.| `fe/fe-core/src/test/java/org/apache/doris/alter/IndexChangeJobTest.java` |Original source hunk header|Disposition| |---|---| |`@@ -30,6 +30,8 @@`|Adapted: Use the target PartitionNamesInfo package and translate operations to AlterClauses. Keep all six source-added cases and verify that partition names survive translation.| |`@@ -806,4 +808,127 @@ public void testCancelNgramBfBuildIndex() throws UserException {`|Adapted: Use the target PartitionNamesInfo package and translate operations to AlterClauses. Keep all six source-added cases and verify that partition names survive translation.| `regression-test/suites/inverted_index_p0/index_change/test_drop_index_on_partition.groovy` |Original source hunk header|Disposition| |---|---| |`@@ -0,0 +1,273 @@`|Adapted: Preserve all ten original partition-drop cases and assertions. On the target, CREATE INDEX registers metadata but does not schedule a physical build, so add explicit build_index_on_table calls before each existing build wait.| Source #69442: `adecd627ab031189821bc3a3cf2fd036aa2426e1` → `ade3f32345c179191ac3329477e386c7be5b3c53` `fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java` |Original source hunk header|Disposition| |---|---| |`@@ -2392,6 +2392,14 @@ public int getAsInt() {`|Adapted: Use DropIndexClause and insert the only-clause guard before target metadata mutations; the source light-drop pre-check context is absent.| `fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java` |Original source hunk header|Disposition| |---|---| |`@@ -1435,6 +1435,56 @@ public void testDropIndexRejectedByPendingBuildOfSameIndex() throws Exception {`|Adapted: Keep all source mixed-clause rejection and metadata/job-count assertions; shorten the explanatory comment.| Source #69443: `8ebb90998a5837a195b067077613b1ef8f62de90` → `a7c700ae87f98fa1744db12be1f1ecffc415da87` `fe/fe-core/src/main/java/org/apache/doris/alter/IndexChangeJob.java` Ported(verbatim): complete file equals original merged source. `fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java` |Original source hunk header|Disposition| |---|---| |`@@ -3888,7 +3888,7 @@ public void buildOrDeleteTableInvertedIndices(Database db, OlapTable olapTable,`|Adapted: Use the shared direction-independent job predicate in the existing partition path. Add the early full-table conflict check required on this branch, which has no source pre-check helper; preserve target light-index type support.| |`@@ -3965,7 +3965,7 @@ private void checkNoIndexChangeJobOnPartitions(Database db, OlapTable olapTable,`|Adapted: Use the shared direction-independent job predicate in the existing partition path. Add the early full-table conflict check required on this branch, which has no source pre-check helper; preserve target light-index type support.| |`@@ -3973,13 +3973,11 @@ && isConflictingIndexChangeJob(indexChangeJob, olapTable.getBaseIndexId(), db.ge`|Adapted: Use the shared direction-independent job predicate in the existing partition path. Add the early full-table conflict check required on this branch, which has no source pre-check helper; preserve target light-index type support.| |`@@ -3987,12 +3985,12 @@ && isConflictingIndexChangeJob(indexChangeJob, originIndexId, dbId, tableId, alt`|Adapted: Use the shared direction-independent job predicate in the existing partition path. Add the early full-table conflict check required on this branch, which has no source pre-check helper; preserve target light-index type support.| `fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java` |Original source hunk header|Disposition| |---|---| |`@@ -1383,6 +1383,58 @@ public void testLightAddAndDropIndexInOneStatementRejected() throws Exception {`|Adapted: Keep full-table and partition conflict assertions, and shorten the unfinished-job comment.| Additional target adapter: analysis/DropIndexClause retains and validates PartitionNamesInfo, renders it in SQL, and exposes it to the legacy handler. The source-added translation test verifies that the partition names survive this adapter. Final target compatibility commit updates the remaining legacy task-submission call to the five-argument direction-independent predicate from #69443. The build/drop direction remains a task execution parameter, not a conflict-filter parameter. The older IndexChangeJobTest fixture also requires an explicit org.apache.doris.common.AnalysisException import for the source validation cases. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
