This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new a5d5a52b3f2 [fix](audit) Fix the audit_log upgrade that never finishes 
in cloud mode (#70393)
a5d5a52b3f2 is described below

commit a5d5a52b3f28d19dc0ffe6fa1d197048a43587e1
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Nov 19 00:01:52 2026 +0800

    [fix](audit) Fix the audit_log upgrade that never finishes in cloud mode 
(#70393)
    
    ### Background
    
    When a frontend becomes master, its `InternalSchemaInitializer` daemon
    thread brings `__internal_schema.audit_log` to the audit schema of its
    version (`InternalSchema.AUDIT_SCHEMA`), retrying every 5 seconds until
    that succeeds. A schema change that only adds value columns is light:
    the FE updates the metadata and nothing is rewritten. Reordering columns
    turns it into a job that rewrites every tablet on a backend, and in
    cloud mode that backend comes from the compute group of the current
    session. Audit events are loaded by column name, so the column order
    does not matter for loading.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: #51363 (introduced the add + reorder upgrade), #52432 (its
    pick to branch-3.1)
    PR Type: Bugfix
    
    Problem Summary:
    
    In cloud mode, a cluster upgraded to a release that adds audit columns
    never gets them: after 4.1.3 -> 4.1.5, for example, `audit_log` has no
    `queue_time_ms`, and the master FE logs `Failed to alter audit table
    schema ... CONNECT_CONTEXT_NOT_SET` every 5 seconds for as long as it
    runs, restarts included. The upgrade was always an add + reorder job,
    which in cloud mode needs a compute group, and the initializer's thread
    has no session to resolve one. This PR adds missing columns in place by
    a light schema change when the table's columns are already in order, as
    in every table created since 3.1, and runs the ALTER in a session of its
    own, so the rewrite a 3.0 table still needs also works in cloud mode.
    Most of the diff is tests.
    
    ### Details
    
    - **Change map:** +60 -18 lines of production code, all in
    `InternalSchemaInitializer`, and +371 -128 of tests (86% of the added
    lines). The code has two parts: (1) `getAuditTableAlterOps` plans the
    ALTER: an `ADD COLUMN ... AFTER` op per missing column when the table's
    columns are in order, the existing add + reorder otherwise (about 50
    lines); (2) `alterAuditSchemaIfNeeded` runs the ALTER inside
    `StatisticsUtil.buildConnectContext(false)` (about 10 lines). The tests:
    planner cases in `InternalSchemaInitializerTest`, both paths on an FE in
    `InternalSchemaAlterTest`, and a new docker suite that restarts the FE
    on an older table.
    - **Frequency:** Always, in cloud mode, when the master FE finds
    `audit_log` short of columns of its schema: after 3.0 -> 3.1/4.0, 4.0 or
    4.1.0-4.1.3 -> 4.1.4+/4.2 (`queue_time_ms`), and 4.1/4.2 -> master
    (`protocol` and two spill columns). Fresh clusters are not affected.
    - **Root cause:** Since #51363, `alterAuditSchemaIfNeeded` always adds a
    `ReorderColumnsOp`, so the change is a rewrite job.
    `SchemaChangeHandler.createJob` asks each replica for its backend; in
    cloud mode `CloudReplica.getBackendId()` ->
    `CloudSystemInfoService.getCurrentClusterId()` throws
    `CONNECT_CONTEXT_NOT_SET` without a `ConnectContext`, which the
    initializer's thread never has. The exception is logged and the loop
    retries forever.
    - **Why existing tests missed it:** Unit tests:
    `InternalSchemaAlterTest` creates the tables fresh, so the upgrade has
    nothing to do; the two audit tests of `InternalSchemaInitializerTest`
    re-implemented the old logic inside the test and never called production
    code; nothing ran the initializer in cloud mode. Regression:
    `audit/test_audit_log_queue_time`, `audit/test_audit_log_protocol` and
    the cloud `audit/test_audit_log_hint_compute_group_docker` all start on
    a fresh audit table; none restarts the FE on an older one.
    - **Fix:** Missing columns follow their predecessor in the schema, in
    one ALTER, when the schema's columns the table has are in order and come
    first: a light change that needs no job, backend or compute group.
    Anything else (a 3.0 table, an unknown column in between) keeps add +
    reorder, now in a session with ADMIN and, in cloud mode, a compute
    group, as `createTable()` of the class already does. The fix belongs
    here: only this daemon thread created a rewrite job without a session.
    - **Other paths and branches:** `modifyColumnStatsTblSchema()` in the
    same thread also alters without a session, but its ops are light on
    every table a cloud-capable version created, so it is unchanged;
    `modifyTblReplicaCount` skips cloud mode. branch-3.1 (#52432),
    branch-4.0, branch-4.1 and branch-4.2 all have the bug.
    - **Behavior change:** Shared-nothing clusters, where the upgrade
    worked, no longer rewrite an audit table that only lacks columns: `SHOW
    ALTER TABLE COLUMN` shows a finished light change (`IndexId` =
    `OriginIndexId`, `TransactionId` -1) instead of a rewrite job. In cloud
    mode, a table that needs reordering is rewritten on the compute group
    resolved for ADMIN, which by the code can start a suspended one.
    
    ### Testing
    
    - FE unit tests, run locally with `run-fe-ut.sh` on this head:
    `InternalSchemaInitializerTest` (8, 5 new, including a 4.1.3 table
    upgraded in place with the schema's types and a 3.0 table reordered),
    `InternalSchemaAlterTest` (5, 2 new, run like the daemon on a thread
    without a session), `AlterJobV2QueryGlobalsTest` (1). Before the fix the
    two new `InternalSchemaAlterTest` cases fail: `expected: <NORMAL> but
    was: <SCHEMA_CHANGE>` and `expected: <'admin'@'%'> but was: <null>`.
    - New docker suite `audit/test_audit_log_schema_upgrade_docker`: drops
    three audit columns and restarts the FE, expecting them back in place
    with the same tablets and filled by new audit rows; then moves `catalog`
    to the end, restarts, and expects the rewrite job to finish. Run locally
    before the rebase onto #63612, which only adds cache invalidation to
    these alter paths, with an FE of this branch and BE/MS 4.1.3 (no master
    BE image for macOS; that FE build turned two new planner features off
    and set `be_exec_version = 10`, not part of this PR):
    
      | Mode | Before the fix | This PR |
      |---|---|---|
    | cloud | fails: `the audit table was not upgraded`; `Failed to alter
    audit table schema ... CONNECT_CONTEXT_NOT_SET` every 5 s | passes, no
    such warning |
    | shared-nothing | fails: the upgrade rewrote the table, all 8 tablets
    replaced | passes; the in-place change also replays on restart |
    
    - No pipeline config in this repository sets `excludeDockerTest =
    false`, so whether CI runs the suite depends on TeamCity, which I could
    not reach to check.
    
    ### Notes for Reviewers
    
    - `alterAuditSchemaIfNeeded` became `@VisibleForTesting public static`,
    like `createTbl` of the class, so a test can run it as the daemon does.
    - The in-place ops build columns through
    `ColumnDef.translateToColumnDefinition()`, as the class's stats-table
    code does, because `AddColumnOp` needs a `ColumnDefinition` for its SQL
    text; the tests check the resulting types against the schema.
    - When picking to a release branch, `testAuditTableOf413AddedInPlace`
    expects master-only columns and needs adjusting.
---
 .../doris/catalog/InternalSchemaInitializer.java   |  78 +++++--
 .../doris/alter/InternalSchemaAlterTest.java       | 126 +++++++++++
 .../catalog/InternalSchemaInitializerTest.java     | 234 ++++++++++-----------
 .../test_audit_log_schema_upgrade_docker.groovy    | 139 ++++++++++++
 4 files changed, 431 insertions(+), 146 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
index ce21e38dd43..410d4deb7a0 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
@@ -20,6 +20,7 @@ package org.apache.doris.catalog;
 import org.apache.doris.analysis.ColumnDef;
 import org.apache.doris.analysis.ColumnNullableType;
 import org.apache.doris.analysis.DbName;
+import org.apache.doris.catalog.info.ColumnPosition;
 import org.apache.doris.catalog.info.TableNameInfo;
 import org.apache.doris.common.Config;
 import org.apache.doris.common.DdlException;
@@ -55,6 +56,7 @@ import com.google.common.collect.Sets;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -583,21 +585,70 @@ public class InternalSchemaInitializer extends Thread {
         return alterAuditSchemaIfNeeded(auditTable);
     }
 
-    private boolean alterAuditSchemaIfNeeded(OlapTable auditTable) {
-        List<ColumnDef> expectedSchema = InternalSchema.AUDIT_SCHEMA;
-        List<String> expectedColumnNames = expectedSchema.stream()
-                .map(ColumnDef::getName)
+    @VisibleForTesting
+    public static boolean alterAuditSchemaIfNeeded(OlapTable auditTable) {
+        List<String> currentColumnNames = auditTable.getBaseSchema().stream()
+                .map(Column::getName)
                 .map(String::toLowerCase)
                 .collect(Collectors.toList());
-        List<Column> currentColumns = auditTable.getBaseSchema();
-        List<String> currentColumnNames = currentColumns.stream()
-                .map(Column::getName)
+        List<AlterTableOp> alterOps = 
getAuditTableAlterOps(currentColumnNames, InternalSchema.AUDIT_SCHEMA);
+        if (alterOps.isEmpty()) {
+            return true;
+        }
+        TableNameInfo auditTableName = new 
TableNameInfo(InternalCatalog.INTERNAL_CATALOG_NAME,
+                FeConstants.INTERNAL_DB_NAME, AuditLoader.AUDIT_LOG_TABLE);
+        AlterTableCommand alterTableCommand = new 
AlterTableCommand(auditTableName, alterOps);
+        // A schema change that rewrites the table creates a replica on a 
backend for every tablet, and in cloud
+        // mode the backend comes from the compute group of the session. This 
daemon thread has no session.
+        try (AutoCloseConnectContext r = 
StatisticsUtil.buildConnectContext(false)) {
+            Env.getCurrentEnv().alterTable(alterTableCommand);
+        } catch (Exception e) {
+            LOG.warn("Failed to alter audit table schema", e);
+            return false;
+        }
+        return true;
+    }
+
+    /**
+     * The ops that bring an audit table whose columns are {@code 
currentColumnNames} (in lower case) to
+     * {@code expectedSchema}, keeping the columns it has beyond the schema at 
the end; none when it is there.
+     *
+     * When the columns the table has of the schema are in the schema's order 
and come before any other column, as
+     * in a table an earlier release created when the releases since only 
added columns, each missing column is
+     * added right after the column before it in the schema: a light schema 
change, which rewrites nothing.
+     * Otherwise the missing columns are appended and the table is reordered, 
which rewrites it.
+     */
+    @VisibleForTesting
+    public static List<AlterTableOp> getAuditTableAlterOps(List<String> 
currentColumnNames,
+            List<ColumnDef> expectedSchema) {
+        List<String> expectedColumnNames = expectedSchema.stream()
+                .map(ColumnDef::getName)
                 .map(String::toLowerCase)
                 .collect(Collectors.toList());
         // check if all expected columns are exists and in the right order
         if (currentColumnNames.size() >= expectedColumnNames.size()
                 && expectedColumnNames.equals(currentColumnNames.subList(0, 
expectedColumnNames.size()))) {
-            return true;
+            return Collections.emptyList();
+        }
+
+        List<String> presentColumnNames = expectedColumnNames.stream()
+                .filter(currentColumnNames::contains)
+                .collect(Collectors.toList());
+        // With the first column of the schema there, every missing column has 
a column before it to follow: one
+        // the table has, or one an earlier op of the same ALTER adds.
+        if (currentColumnNames.contains(expectedColumnNames.get(0))
+                && presentColumnNames.equals(currentColumnNames.subList(0, 
presentColumnNames.size()))) {
+            List<AlterTableOp> addColumnOps = Lists.newArrayList();
+            for (int i = 1; i < expectedSchema.size(); i++) {
+                if (!presentColumnNames.contains(expectedColumnNames.get(i))) {
+                    ColumnDefinition columnDefinition = 
expectedSchema.get(i).translateToColumnDefinition();
+                    AddColumnOp addColumnOp = new AddColumnOp(columnDefinition,
+                            new ColumnPosition(expectedSchema.get(i - 
1).getName()), null, null);
+                    
addColumnOp.setColumn(columnDefinition.translateToCatalogStyleForSchemaChange());
+                    addColumnOps.add(addColumnOp);
+                }
+            }
+            return addColumnOps;
         }
 
         List<AlterTableOp> alterClauses = Lists.newArrayList();
@@ -619,16 +670,7 @@ public class InternalSchemaInitializer extends Thread {
         newColumnOrders.addAll(removedColumnNames);
         ReorderColumnsOp reorderColumnsOp = new 
ReorderColumnsOp(newColumnOrders, null, Maps.newHashMap());
         alterClauses.add(reorderColumnsOp);
-        TableNameInfo auditTableName = new 
TableNameInfo(InternalCatalog.INTERNAL_CATALOG_NAME,
-                FeConstants.INTERNAL_DB_NAME, AuditLoader.AUDIT_LOG_TABLE);
-        AlterTableCommand alterTableCommand = new 
AlterTableCommand(auditTableName, alterClauses);
-        try {
-            Env.getCurrentEnv().alterTable(alterTableCommand);
-        } catch (Exception e) {
-            LOG.warn("Failed to alter audit table schema", e);
-            return false;
-        }
-        return true;
+        return alterClauses;
     }
 
     public static boolean isStatsTableSchemaValid() {
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java
index af4901864d9..c210ab37437 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java
@@ -18,25 +18,37 @@
 package org.apache.doris.alter;
 
 import org.apache.doris.analysis.ColumnDef;
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Column;
 import org.apache.doris.catalog.Database;
 import org.apache.doris.catalog.Env;
 import org.apache.doris.catalog.InternalSchema;
 import org.apache.doris.catalog.InternalSchemaInitializer;
 import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.OlapTable.OlapTableState;
 import org.apache.doris.catalog.Partition;
 import org.apache.doris.catalog.PartitionInfo;
 import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.Tablet;
 import org.apache.doris.common.AnalysisException;
 import org.apache.doris.common.Config;
 import org.apache.doris.common.FeConstants;
 import org.apache.doris.common.util.PropertyAnalyzer;
 import org.apache.doris.plugin.audit.AuditLoader;
+import org.apache.doris.qe.ConnectContext;
 import org.apache.doris.statistics.StatisticConstants;
 import org.apache.doris.utframe.TestWithFeService;
 
+import com.google.common.collect.Lists;
 import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.Test;
 
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
 public class InternalSchemaAlterTest extends TestWithFeService {
 
     @Override
@@ -44,6 +56,12 @@ public class InternalSchemaAlterTest extends 
TestWithFeService {
         return 3;
     }
 
+    @Override
+    protected void beforeCreatingConnectContext() throws Exception {
+        // the schema change handler takes this interval when the Env is 
built; a job then finishes in seconds
+        FeConstants.default_scheduler_interval_millisecond = 100;
+    }
+
     @Override
     protected void runBeforeAll() throws Exception {
         Config.allow_replica_on_same_host = true;
@@ -99,4 +117,112 @@ public class InternalSchemaAlterTest extends 
TestWithFeService {
         OlapTable table = 
db.getOlapTableOrAnalysisException(StatisticConstants.PARTITION_STATISTIC_TBL_NAME);
         Assertions.assertEquals(PrimitiveType.HLL, 
table.getColumn("ndv").getType().getPrimitiveType());
     }
+
+    // A table an earlier version created lacks the columns added since, and 
has the others in order. They are
+    // added in place, by a light schema change: no job to wait for, and the 
table keeps its index and tablets.
+    @Test
+    public void testAuditTableMissingColumnsAddedInPlace() throws Exception {
+        OlapTable table = auditTable();
+        try {
+            alterTableSync("ALTER TABLE __internal_schema.audit_log"
+                    + " DROP COLUMN queue_time_ms, DROP COLUMN cpu_time_ms, 
DROP COLUMN compute_group");
+            Assertions.assertNull(table.getColumn("queue_time_ms"));
+            long baseIndexId = table.getBaseIndexId();
+            Set<Long> tabletIds = baseTabletIds(table);
+
+            Assertions.assertTrue(alterAuditSchemaAsInitializer(table));
+
+            Assertions.assertEquals(OlapTableState.NORMAL, table.getState());
+            Assertions.assertEquals(auditSchemaColumnNames(), 
baseColumnNames(table));
+            Assertions.assertEquals(baseIndexId, table.getBaseIndexId());
+            Assertions.assertEquals(tabletIds, baseTabletIds(table));
+            for (String columnName : Lists.newArrayList("queue_time_ms", 
"cpu_time_ms", "compute_group")) {
+                ColumnDef columnDef = InternalSchema.AUDIT_SCHEMA.stream()
+                        .filter(def -> 
def.getName().equals(columnName)).findFirst().get();
+                Column column = table.getColumn(columnName);
+                Assertions.assertEquals(columnDef.getType(), column.getType(), 
columnName);
+                Assertions.assertEquals(columnDef.isAllowNull(), 
column.isAllowNull(), columnName);
+            }
+        } finally {
+            waitAlterJobsDone(table);
+        }
+    }
+
+    // A table whose columns are out of order needs the add + reorder job, 
which rewrites it. The initializer
+    // creates the job in a session of its own: in cloud mode the job picks 
its backends from the compute group
+    // of the session, and the initializer's daemon thread has no session.
+    @Test
+    public void testAuditTableOutOfOrderReorderedInOwnSession() throws 
Exception {
+        OlapTable table = auditTable();
+        try {
+            String catalogType = table.getColumn("catalog").getType().toSql();
+            alterTableSync("ALTER TABLE __internal_schema.audit_log DROP 
COLUMN catalog, DROP COLUMN queue_time_ms");
+            alterTableSync("ALTER TABLE __internal_schema.audit_log ADD COLUMN 
catalog " + catalogType + " NULL");
+            List<String> columnNames = baseColumnNames(table);
+            Assertions.assertEquals("catalog", 
columnNames.get(columnNames.size() - 1));
+            Set<Long> jobIds = 
schemaChangeJobs(table).stream().map(AlterJobV2::getJobId)
+                    .collect(Collectors.toSet());
+
+            Assertions.assertTrue(alterAuditSchemaAsInitializer(table));
+
+            List<AlterJobV2> newJobs = schemaChangeJobs(table).stream()
+                    .filter(job -> 
!jobIds.contains(job.getJobId())).collect(Collectors.toList());
+            Assertions.assertEquals(1, newJobs.size());
+            Assertions.assertEquals(UserIdentity.ADMIN, 
newJobs.get(0).userIdentity);
+            waitAlterJobsDone(table);
+            Assertions.assertEquals(AlterJobV2.JobState.FINISHED, 
newJobs.get(0).getJobState());
+            Assertions.assertEquals(auditSchemaColumnNames(), 
baseColumnNames(table));
+        } finally {
+            waitAlterJobsDone(table);
+        }
+    }
+
+    private static OlapTable auditTable() throws AnalysisException {
+        return 
Env.getCurrentEnv().getCatalogMgr().getInternalCatalog().getDbNullable(FeConstants.INTERNAL_DB_NAME)
+                .getOlapTableOrAnalysisException(AuditLoader.AUDIT_LOG_TABLE);
+    }
+
+    // Runs alterAuditSchemaIfNeeded the way InternalSchemaInitializer does: 
on a thread of its own, which
+    // has no session.
+    private static boolean alterAuditSchemaAsInitializer(OlapTable table) 
throws Exception {
+        FutureTask<Boolean> task = new FutureTask<>(() -> {
+            Assertions.assertNull(ConnectContext.get());
+            boolean altered = 
InternalSchemaInitializer.alterAuditSchemaIfNeeded(table);
+            Assertions.assertNull(ConnectContext.get(), "the thread keeps the 
session built for the alter");
+            return altered;
+        });
+        new Thread(task, "InternalSchemaInitializer").start();
+        return task.get(60, TimeUnit.SECONDS);
+    }
+
+    private static List<String> auditSchemaColumnNames() {
+        return 
InternalSchema.AUDIT_SCHEMA.stream().map(ColumnDef::getName).collect(Collectors.toList());
+    }
+
+    private static List<String> baseColumnNames(OlapTable table) {
+        return 
table.getBaseSchema().stream().map(Column::getName).collect(Collectors.toList());
+    }
+
+    private static Set<Long> baseTabletIds(OlapTable table) {
+        return table.getPartitions().stream().flatMap(partition -> 
partition.getBaseIndex().getTablets().stream())
+                .map(Tablet::getId).collect(Collectors.toSet());
+    }
+
+    private static List<AlterJobV2> schemaChangeJobs(OlapTable table) {
+        return 
Env.getCurrentEnv().getSchemaChangeHandler().getAlterJobsV2().values().stream()
+                .filter(job -> job.getTableId() == 
table.getId()).collect(Collectors.toList());
+    }
+
+    private static void waitAlterJobsDone(OlapTable table) throws 
InterruptedException {
+        for (AlterJobV2 job : schemaChangeJobs(table)) {
+            for (int i = 0; i < 600 && !job.getJobState().isFinalState(); i++) 
{
+                Thread.sleep(100);
+            }
+            Assertions.assertTrue(job.getJobState().isFinalState(), 
job.toString());
+        }
+        for (int i = 0; i < 600 && table.getState() != OlapTableState.NORMAL; 
i++) {
+            Thread.sleep(100);
+        }
+        Assertions.assertEquals(OlapTableState.NORMAL, table.getState());
+    }
 }
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
 
b/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
index 9584aa034aa..9dc48403be6 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
@@ -18,21 +18,25 @@
 package org.apache.doris.catalog;
 
 import org.apache.doris.analysis.ColumnDef;
-import org.apache.doris.catalog.info.ColumnPosition;
+import org.apache.doris.analysis.ColumnNullableType;
 import org.apache.doris.common.UserException;
-import org.apache.doris.nereids.trees.plans.commands.info.AlterOp;
+import org.apache.doris.nereids.trees.plans.commands.info.AddColumnOp;
+import org.apache.doris.nereids.trees.plans.commands.info.AddColumnsOp;
 import org.apache.doris.nereids.trees.plans.commands.info.AlterTableOp;
 import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnOp;
+import org.apache.doris.nereids.trees.plans.commands.info.ReorderColumnsOp;
 import org.apache.doris.plugin.audit.AuditLoader;
 import org.apache.doris.statistics.StatisticConstants;
 
 import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
 import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.mockito.Mockito;
 
+import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
+import java.util.stream.Collectors;
 
 class InternalSchemaInitializerTest {
     @Test
@@ -123,143 +127,117 @@ class InternalSchemaInitializerTest {
                 "scan_bytes_from_remote_storage field is missing from the 
copied schema");
     }
 
+    // An up-to-date table, also one with columns beyond the schema after it, 
needs nothing.
     @Test
-    public void testStorageColumnsPositionInAuditTable() throws Exception {
-        // Get storage-related column definitions directly from 
InternalSchema.AUDIT_SCHEMA
-        ColumnDef localStorageDef = null;
-        ColumnDef remoteStorageDef = null;
-
-        for (int i = 0; i < InternalSchema.AUDIT_SCHEMA.size(); i++) {
-            ColumnDef def = InternalSchema.AUDIT_SCHEMA.get(i);
-            if (def.getName().equals("scan_bytes_from_local_storage")) {
-                localStorageDef = def;
-            } else if (def.getName().equals("scan_bytes_from_remote_storage")) 
{
-                remoteStorageDef = def;
-            }
-        }
-
-        Assertions.assertNotNull(localStorageDef, "The 
scan_bytes_from_local_storage column should exist in AUDIT_SCHEMA");
-        Assertions.assertNotNull(remoteStorageDef, "The 
scan_bytes_from_remote_storage column should exist in AUDIT_SCHEMA");
-
-        // Simulate column position logic in InternalSchemaInitializer
-        // Note: Based on test failure, the system uses FIRST position rather 
than after a specific column
-        List<AlterOp> alterOps = Lists.newArrayList();
-
-        // Add scan_bytes_from_local_storage column using FIRST position
-        ColumnPosition localStoragePosition = ColumnPosition.FIRST;
-        ModifyColumnOp modifyColumnOp = new ModifyColumnOp(
-                localStorageDef.translateToColumnDefinition(), 
localStoragePosition, null, Maps.newHashMap());
-        modifyColumnOp.setColumn(localStorageDef.toColumn());
-        alterOps.add(modifyColumnOp);
-
-        // Add scan_bytes_from_remote_storage column using FIRST position
-        ColumnPosition remoteStoragePosition = ColumnPosition.FIRST;
-        ModifyColumnOp modifyColumnOp1 = new ModifyColumnOp(
-                remoteStorageDef.translateToColumnDefinition(), 
remoteStoragePosition, null, Maps.newHashMap());
-        modifyColumnOp1.setColumn(remoteStorageDef.toColumn());
-        alterOps.add(modifyColumnOp1);
-
-        // Verify the generated AlterClauses
-        Assertions.assertEquals(2, alterOps.size(), "Two AlterClauses should 
be generated");
+    public void testAuditTableUpToDate() {
+        List<ColumnDef> schema = schema("a", "b", "c");
+        Assertions.assertEquals(Collections.emptyList(), 
describe(alterOps(schema, "a", "b", "c")));
+        Assertions.assertEquals(Collections.emptyList(), 
describe(alterOps(schema, "a", "b", "c", "x")));
+    }
 
-        // Verify that column positions are FIRST
-        Assertions.assertTrue(((ModifyColumnOp) 
alterOps.get(0)).getColPos().isFirst(),
-                "The position of the scan_bytes_from_local_storage column 
should be FIRST");
-        Assertions.assertTrue(((ModifyColumnOp) 
alterOps.get(1)).getColPos().isFirst(),
-                "The position of the scan_bytes_from_remote_storage column 
should be FIRST");
+    // The columns the table has of the schema are in order and come first: 
each missing column is added right
+    // after the column before it in the schema, by a light schema change.
+    @Test
+    public void testAuditTableMissingColumnsAddedInPlace() {
+        List<ColumnDef> schema = schema("a", "b", "c", "d", "e");
+        Assertions.assertEquals(Lists.newArrayList("ADD b AFTER a", "ADD d 
AFTER c"),
+                describe(alterOps(schema, "a", "c", "e")));
+        Assertions.assertEquals(Lists.newArrayList("ADD e AFTER d"), 
describe(alterOps(schema, "a", "b", "c", "d")));
+        // the second of two adjacent columns follows the first, which the 
same ALTER adds
+        Assertions.assertEquals(Lists.newArrayList("ADD b AFTER a", "ADD c 
AFTER b"),
+                describe(alterOps(schema, "a", "d", "e")));
+        // a column beyond the schema stays at the end
+        Assertions.assertEquals(Lists.newArrayList("ADD b AFTER a", "ADD d 
AFTER c"),
+                describe(alterOps(schema, "a", "c", "e", "x")));
     }
 
+    // Any other table gets the missing columns appended and is reordered, 
which rewrites it.
     @Test
-    public void testDoesNotModifyExistingColumns() throws Exception {
-        // Create a mock audit table with storage-related columns but with 
inconsistent types (VARCHAR instead of BIGINT)
-        List<Column> initialSchema = Lists.newArrayList(
-                new Column("query_id", ScalarType.createVarcharType(48), true, 
null, false, null, ""),
-                new Column("time", ScalarType.createDatetimeV2Type(3), true, 
null, false, null, ""),
-                new Column("client_ip", ScalarType.createVarcharType(128), 
true, null, false, null, ""),
-                new Column("user", ScalarType.createVarcharType(128), true, 
null, false, null, ""),
-                new Column("catalog", ScalarType.createVarcharType(128), true, 
null, false, null, ""),
-                new Column("db", ScalarType.createVarcharType(128), true, 
null, false, null, ""),
-                new Column("state", ScalarType.createVarcharType(128), true, 
null, false, null, ""),
-                new Column("error_code", ScalarType.INT, true, null, false, 
null, ""),
-                new Column("error_message", ScalarType.STRING, true, null, 
false, null, ""),
-                new Column("query_time", ScalarType.BIGINT, true, null, false, 
null, ""),
-                new Column("scan_bytes", ScalarType.BIGINT, true, null, false, 
null, ""),
-                new Column("scan_rows", ScalarType.BIGINT, true, null, false, 
null, ""),
-                new Column("return_rows", ScalarType.BIGINT, true, null, 
false, null, ""),
-                new Column("shuffle_send_rows", ScalarType.BIGINT, true, null, 
false, null, ""),
-                new Column("shuffle_send_bytes", ScalarType.BIGINT, true, 
null, false, null, ""),
-                // Intentionally use inconsistent types (VARCHAR instead of 
BIGINT)
-                new Column("scan_bytes_from_local_storage", 
ScalarType.createVarcharType(128), true, null, false, null, ""),
-                new Column("scan_bytes_from_remote_storage", 
ScalarType.createVarcharType(128), true, null, false, null, "")
-        );
+    public void testAuditTableOutOfOrderReordered() {
+        List<ColumnDef> schema = schema("a", "b", "c", "d", "e");
+        // two columns of the schema swapped
+        Assertions.assertEquals(Lists.newArrayList("ADD (d)", "ORDER BY (a, b, 
c, d, e)"),
+                describe(alterOps(schema, "a", "c", "b", "e")));
+        Assertions.assertEquals(Lists.newArrayList("ORDER BY (a, b, c, d, e)"),
+                describe(alterOps(schema, "b", "a", "c", "d", "e")));
+        // a column beyond the schema between two of it
+        Assertions.assertEquals(Lists.newArrayList("ADD (b)", "ORDER BY (a, b, 
c, d, e, x)"),
+                describe(alterOps(schema, "a", "x", "c", "d", "e")));
+        // the first column missing: there is no column for it to follow
+        Assertions.assertEquals(Lists.newArrayList("ADD (a)", "ORDER BY (a, b, 
c, d, e)"),
+                describe(alterOps(schema, "b", "c", "d", "e")));
+    }
 
-        // Use the correct constructor to create OlapTable to ensure 
nameToColumn is properly initialized
-        OlapTable auditTable = new OlapTable(1000, "audit_log", initialSchema, 
KeysType.AGG_KEYS,
-                new SinglePartitionInfo(), new HashDistributionInfo());
+    // A table 4.1.3 created lacks the four columns added since. Each is added 
in place, as the audit schema
+    // defines it.
+    @Test
+    public void testAuditTableOf413AddedInPlace() {
+        List<String> added = Lists.newArrayList("queue_time_ms", 
"spill_write_bytes_to_remote_storage",
+                "spill_read_bytes_from_remote_storage", "protocol");
+        List<String> currentColumnNames = auditColumnNames();
+        currentColumnNames.removeAll(added);
+
+        List<AlterTableOp> ops = 
InternalSchemaInitializer.getAuditTableAlterOps(currentColumnNames,
+                InternalSchema.AUDIT_SCHEMA);
+
+        Assertions.assertEquals(Lists.newArrayList("ADD queue_time_ms AFTER 
query_time",
+                "ADD spill_write_bytes_to_remote_storage AFTER 
spill_read_bytes_from_local_storage",
+                "ADD spill_read_bytes_from_remote_storage AFTER 
spill_write_bytes_to_remote_storage",
+                "ADD protocol AFTER compute_group"), describe(ops));
+        for (AlterTableOp op : ops) {
+            Column column = ((AddColumnOp) op).getColumn();
+            ColumnDef columnDef = InternalSchema.AUDIT_SCHEMA.stream()
+                    .filter(def -> 
def.getName().equals(column.getName())).findFirst().get();
+            Assertions.assertEquals(columnDef.getType(), column.getType());
+            Assertions.assertEquals(columnDef.isAllowNull(), 
column.isAllowNull());
+            Assertions.assertFalse(column.isKey());
+        }
+    }
 
-        // Verify columns exist and have VARCHAR type
-        Column localStorageCol = 
auditTable.getColumn("scan_bytes_from_local_storage");
-        Column remoteStorageCol = 
auditTable.getColumn("scan_bytes_from_remote_storage");
+    // A table 3.0 created has its columns in the order the audit schema had 
before #51363 moved them around. The
+    // missing columns are appended and the table is reordered.
+    @Test
+    public void testAuditTableOf30Reordered() {
+        List<String> currentColumnNames = Lists.newArrayList("query_id", 
"time", "client_ip", "user", "catalog", "db",
+                "state", "error_code", "error_message", "query_time", 
"scan_bytes", "scan_rows", "return_rows",
+                "shuffle_send_rows", "shuffle_send_bytes", 
"scan_bytes_from_local_storage",
+                "scan_bytes_from_remote_storage", "stmt_id", "stmt_type", 
"is_query", "is_nereids", "frontend_ip",
+                "cpu_time_ms", "sql_hash", "sql_digest", "peak_memory_bytes", 
"workload_group", "compute_group",
+                "stmt");
+        List<String> missing = auditColumnNames();
+        missing.removeAll(currentColumnNames);
+
+        List<AlterTableOp> ops = 
InternalSchemaInitializer.getAuditTableAlterOps(currentColumnNames,
+                InternalSchema.AUDIT_SCHEMA);
+
+        Assertions.assertEquals(Lists.newArrayList("ADD (" + String.join(", ", 
missing) + ")",
+                "ORDER BY (" + String.join(", ", auditColumnNames()) + ")"), 
describe(ops));
+    }
 
-        Assertions.assertNotNull(localStorageCol, "The 
scan_bytes_from_local_storage column should exist in auditTable");
-        Assertions.assertNotNull(remoteStorageCol, "The 
scan_bytes_from_remote_storage column should exist in auditTable");
-        Assertions.assertTrue(localStorageCol.getType().isVarchar(),
-                "The scan_bytes_from_local_storage column type should be 
VARCHAR");
-        Assertions.assertTrue(remoteStorageCol.getType().isVarchar(),
-                "The scan_bytes_from_remote_storage column type should be 
VARCHAR");
+    private static List<ColumnDef> schema(String... columnNames) {
+        return Arrays.stream(columnNames).map(name -> new ColumnDef(name, 
ScalarType.createType(PrimitiveType.BIGINT),
+                ColumnNullableType.NULLABLE)).collect(Collectors.toList());
+    }
 
-        // Get complete column definitions from InternalSchema.AUDIT_SCHEMA
-        List<ColumnDef> expectedSchema = Lists.newArrayList();
-        for (ColumnDef def : InternalSchema.AUDIT_SCHEMA) {
-            expectedSchema.add(def);
-        }
+    private static List<AlterTableOp> alterOps(List<ColumnDef> schema, 
String... currentColumnNames) {
+        return 
InternalSchemaInitializer.getAuditTableAlterOps(Lists.newArrayList(currentColumnNames),
 schema);
+    }
 
-        // Simulate column processing logic in InternalSchemaInitializer
-        List<AlterOp> alterOps = Lists.newArrayList();
+    private static List<String> auditColumnNames() {
+        return 
InternalSchema.AUDIT_SCHEMA.stream().map(ColumnDef::getName).collect(Collectors.toList());
+    }
 
-        // Add columns if they don't exist
-        for (int i = 0; i < expectedSchema.size(); i++) {
-            ColumnDef def = expectedSchema.get(i);
-            if (auditTable.getColumn(def.getName()) == null) {
-                // If column doesn't exist, add it
-                String afterColumn = null;
-                if (i > 0) {
-                    for (int j = i - 1; j >= 0; j--) {
-                        String prevColName = expectedSchema.get(j).getName();
-                        if (auditTable.getColumn(prevColName) != null) {
-                            afterColumn = prevColName;
-                            break;
-                        }
-                    }
-                }
-                ColumnPosition position = afterColumn == null ? 
ColumnPosition.FIRST :
-                        new ColumnPosition(afterColumn);
-                ModifyColumnOp op = new 
ModifyColumnOp(def.translateToColumnDefinition(), position, null, 
Maps.newHashMap());
-                op.setColumn(def.toColumn());
-                alterOps.add(op);
+    private static List<String> describe(List<AlterTableOp> ops) {
+        return ops.stream().map(op -> {
+            if (op instanceof AddColumnOp) {
+                AddColumnOp addColumnOp = (AddColumnOp) op;
+                return "ADD " + addColumnOp.getColumn().getName() + " AFTER " 
+ addColumnOp.getColPos().getLastCol();
             }
-            // Note: InternalSchemaInitializer.created() method does not check 
if column types match
-            // It only adds columns that don't exist in the table
-        }
-
-        // Check if AlterClauses were generated for storage-related columns
-        boolean hasLocalStorageClause = false;
-        boolean hasRemoteStorageClause = false;
-
-        for (AlterOp op : alterOps) {
-            ModifyColumnOp modifyColumnOp = (ModifyColumnOp) op;
-            if 
(modifyColumnOp.getColumn().getName().equals("scan_bytes_from_local_storage")) {
-                hasLocalStorageClause = true;
-            } else if 
(modifyColumnOp.getColumn().getName().equals("scan_bytes_from_remote_storage")) 
{
-                hasRemoteStorageClause = true;
+            if (op instanceof AddColumnsOp) {
+                return "ADD (" + ((AddColumnsOp) 
op).getColumns().stream().map(Column::getName)
+                        .collect(Collectors.joining(", ")) + ")";
             }
-        }
-
-        // Verify the system does not generate AlterClauses for columns that 
already exist
-        // even if their types don't match the expected types
-        Assertions.assertFalse(hasLocalStorageClause,
-                "The system should not generate AlterClause for the 
scan_bytes_from_local_storage column that already exists");
-        Assertions.assertFalse(hasRemoteStorageClause,
-                "The system should not generate AlterClause for the 
scan_bytes_from_remote_storage column that already exists");
+            return "ORDER BY (" + String.join(", ", ((ReorderColumnsOp) 
op).getColumnsByPos()) + ")";
+        }).collect(Collectors.toList());
     }
 }
diff --git 
a/regression-test/suites/audit/test_audit_log_schema_upgrade_docker.groovy 
b/regression-test/suites/audit/test_audit_log_schema_upgrade_docker.groovy
new file mode 100644
index 00000000000..a72d5dd7df4
--- /dev/null
+++ b/regression-test/suites/audit/test_audit_log_schema_upgrade_docker.groovy
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+// When a frontend becomes master, InternalSchemaInitializer brings 
__internal_schema.audit_log to the
+// audit schema of its version, so a table an earlier version created gains 
the columns added since.
+// A table that only lacks columns gets them in place, by a light schema 
change that rewrites nothing.
+// A table whose columns are out of order is rewritten by an add + reorder 
job, which in cloud mode has
+// to run on a compute group: the initializer's daemon thread used to have 
none, and the upgrade failed
+// with CONNECT_CONTEXT_NOT_SET on every retry.
+suite("test_audit_log_schema_upgrade_docker", "docker") {
+    def options = new ClusterOptions()
+    options.feNum = 1
+    options.beNum = 1
+    // Both modes: the in-place add applies to either, and the rewrite failed 
in cloud mode.
+    options.cloudMode = null
+
+    docker(options) {
+        def columnNames = {
+            return (sql "desc __internal_schema.audit_log").collect { it[0] as 
String }
+        }
+        // the name, type and nullability of each column, in order
+        def columns = {
+            return (sql "desc __internal_schema.audit_log").collect { 
"${it[0]} ${it[1]} ${it[2]}".toString() }
+        }
+        def tabletIds = {
+            return (sql_return_maparray "show tablets from 
__internal_schema.audit_log")
+                    .collect { it.TabletId as String } as Set
+        }
+        def latestAlterJob = {
+            def jobs = sql_return_maparray """SHOW ALTER TABLE COLUMN FROM 
__internal_schema
+                    WHERE TableName = 'audit_log' ORDER BY CreateTime DESC 
LIMIT 1"""
+            return jobs.isEmpty() ? null : jobs[0]
+        }
+
+        // The first master creates the table with the full schema: the shape 
an upgrade must reach.
+        def expectedColumns = null
+        for (int i = 0; i < 60 && expectedColumns == null; i++) {
+            try {
+                expectedColumns = columns()
+            } catch (Exception e) {
+                logger.info("the audit table is not created yet: 
${e.getMessage()}")
+                sleep(1000)
+            }
+        }
+        assertNotNull(expectedColumns)
+        assertTrue(columnNames().containsAll(["catalog", "query_time", 
"queue_time_ms", "cpu_time_ms",
+                "compute_group"]), "${expectedColumns}")
+
+        // A restarted frontend becomes master again and runs 
InternalSchemaInitializer against the
+        // table as it is now.
+        def restartFrontend = {
+            cluster.restartFrontends()
+            for (int i = 0; i < 60; i++) {
+                try {
+                    context.reconnectFe()
+                    sql "show frontends"
+                    return
+                } catch (Exception e) {
+                    logger.info("the restarted frontend is not ready yet: 
${e.getMessage()}")
+                    sleep(1000)
+                }
+            }
+            throw new IllegalStateException("cannot connect to the restarted 
frontend")
+        }
+        def waitForAuditSchema = { int timeoutSeconds ->
+            def actual = columns()
+            for (int i = 0; i < timeoutSeconds && actual != expectedColumns; 
i++) {
+                sleep(1000)
+                actual = columns()
+            }
+            assertEquals(expectedColumns, actual,
+                    "the audit table was not upgraded; latest schema change 
job: ${latestAlterJob()}")
+        }
+
+        // 1. A table of an earlier version lacks the columns added since and 
has the others in order;
+        // queue_time_ms, for one, came with 4.1.4. Drop it, the column after 
it, and one further on.
+        sql """ALTER TABLE __internal_schema.audit_log
+                DROP COLUMN queue_time_ms, DROP COLUMN cpu_time_ms, DROP 
COLUMN compute_group"""
+        assertFalse(columnNames().contains("queue_time_ms"))
+        def tabletsBefore = tabletIds()
+
+        restartFrontend()
+        waitForAuditSchema(120)
+        // Added in place: the table keeps its tablets.
+        def tabletsAfter = tabletIds()
+        assertTrue(tabletsAfter.containsAll(tabletsBefore), "tablets before: 
${tabletsBefore}, after: ${tabletsAfter}")
+
+        // New audit rows fill the columns the upgrade added.
+        sql "set global enable_audit_plugin = true"
+        def marker = "audit_schema_upgrade_marker_5E1C9A"
+        sql "select 1 as ${marker}"
+        def countMarkedRows = {
+            return (sql """select count(*) from __internal_schema.audit_log
+                    where stmt like '%${marker}%' and stmt not like 
'%audit_log%'
+                      and queue_time_ms is not null and cpu_time_ms is not 
null""")[0][0] as long
+        }
+        def found = countMarkedRows()
+        for (int i = 0; i < 60 && found == 0; i++) {
+            sleep(3000)
+            sql "call flush_audit_log()"
+            found = countMarkedRows()
+        }
+        assertTrue(found >= 1, "no audit row of the marked statement fills the 
added columns")
+
+        // 2. A table created before #51363 moved the audit columns around has 
them out of order. Move
+        // catalog to the end and drop queue_time_ms: the upgrade has to 
rewrite the table, which in
+        // cloud mode runs on a compute group.
+        def catalogType = (sql "desc __internal_schema.audit_log").find { 
it[0] == "catalog" }[1]
+        sql "ALTER TABLE __internal_schema.audit_log DROP COLUMN catalog, DROP 
COLUMN queue_time_ms"
+        sql "ALTER TABLE __internal_schema.audit_log ADD COLUMN catalog 
${catalogType} NULL"
+        assertEquals("catalog", columnNames().last())
+        tabletsBefore = tabletIds()
+
+        restartFrontend()
+        waitForAuditSchema(300)
+        def job = latestAlterJob()
+        assertEquals("FINISHED", job.State, "${job}")
+        // Rewritten: none of the old tablets is left.
+        tabletsAfter = tabletIds()
+        assertTrue(tabletsAfter.intersect(tabletsBefore).isEmpty(),
+                "tablets before: ${tabletsBefore}, after: ${tabletsAfter}")
+    }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to