This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e4e63a7a39c [refactor](catalog) remove per-data-source legacy types
from fe-core (#66303)
e4e63a7a39c is described below
commit e4e63a7a39cd3ddf95180fc69d13c4eca0a318a1
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Jul 31 17:40:53 2026 +0800
[refactor](catalog) remove per-data-source legacy types from fe-core
(#66303)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #64304
Problem Summary:
Follow-up cleanup for #64304. After external catalogs moved behind the
connector plugin SPI, fe-core still declared a name for every data
source it no longer knows anything about. This removes those names:
- **`TableIf.TableType`**: the per-source `*_EXTERNAL_TABLE` constants
(`HMS`, `ES`, `JDBC`, `ICEBERG`, `PAIMON`, `MAX_COMPUTE`, `HUDI`,
`TRINO_CONNECTOR`, `LAKESOUl`) and the deprecated internal-catalog
`ICEBERG` / `HUDI`. An external table served by a connector plugin is a
`PLUGIN_EXTERNAL_TABLE`; the source's own name is answered by
`PluginDrivenExternalCatalog#getDisplayEngineName`, never by a mapping
held in fe-core.
- **`TableFormatType`**: `hive`, `iceberg`, `hudi`, `paimon`,
`max_compute`, `transactional_hive`, `lakesoul`, `trino_connector`. A
connector names its own format string through
`ConnectorScanRange#getTableFormatType()`, which `PluginDrivenScanNode`
forwards to BE verbatim. Only `tvf` and `remote_doris` are still
produced by fe-core itself. The enum is not persisted anywhere — it only
builds the thrift string.
- **`InitDatabaseLog.Type`**, and with it `ExternalDatabase#dbLogType`,
a field that was assigned and never read. `OP_INIT_EXTERNAL_DB` has been
ignored on replay since 4.0 and `EditLog#logInitExternalDb` has no
callers, so nothing writes one any more; the payload class stays only so
`JournalEntity` can still consume such an entry out of an old journal.
- **`InitCatalogLog.Type.HUDI`**, which was never produced: hudi tables
have always lived in an hms catalog and no `HudiExternalCatalog` ever
existed.
- **`Database#discardHudiTable`**, dead once `TableType.HUDI` is gone.
The remaining `InitCatalogLog.Type` values are kept on purpose. That
enum is also the type of the persisted `ExternalCatalog#logType` field,
which `PluginDrivenExternalCatalog#gsonPostProcess` reads to backfill
the catalog type for the resource-backed catalogs (`es`, `jdbc`) that
never persisted one. Deleting a name there would make it deserialize to
`null` and lose that catalog's type on upgrade.
**Upgrade compatibility.** An image written before the cutover still
carries the old `TableType` names. It stays readable because the
persisted table class is remapped by `GsonUtils`' compatible-subtype
registry, and the stale `type` string deserializes to `null` — GSON
returns `null` for an enum name it does not know rather than throwing —
which `PluginDrivenExternalTable#gsonPostProcess` then normalizes to
`PLUGIN_EXTERNAL_TABLE`, the same normalization it already applied to a
recognized legacy name.
Nothing in the compiler enforces that two-step property, and a
regression would surface not as a build error but as an FE replaying a
persisted external table with a `null` type. So the new
`LegacyExternalTableTypeReplayTest` pins it for every deleted name,
deliberately spelled as string literals so the constants cannot come
back just to keep a test compiling.
---
.../doris/connector/iceberg/IcebergScanRange.java | 4 +-
.../connector/iceberg/IcebergScanRangeTest.java | 4 +-
.../main/java/org/apache/doris/alter/Alter.java | 7 --
.../java/org/apache/doris/catalog/Database.java | 16 ----
.../java/org/apache/doris/catalog/TableIf.java | 47 +++++-----
.../doris/common/cache/NereidsSqlCacheManager.java | 1 -
.../apache/doris/datasource/ExternalDatabase.java | 6 +-
.../doris/RemoteDorisExternalDatabase.java | 3 +-
.../infoschema/ExternalInfoSchemaDatabase.java | 3 +-
.../infoschema/ExternalMysqlDatabase.java | 3 +-
.../doris/datasource/log/InitCatalogLog.java | 12 ++-
.../doris/datasource/log/InitDatabaseLog.java | 29 ++-----
.../plugin/PluginDrivenExternalDatabase.java | 5 +-
.../doris/datasource/scan/TableFormatType.java | 15 ++--
.../datasource/test/TestExternalDatabase.java | 3 +-
.../doris/nereids/rules/analysis/BindRelation.java | 4 -
.../plans/commands/ShowPartitionsCommand.java | 4 +-
.../PartitionValuesTableValuedFunction.java | 1 -
.../PartitionsTableValuedFunction.java | 1 -
.../LegacyExternalTableTypeReplayTest.java | 99 ++++++++++++++++++++++
.../doris/datasource/PluginDrivenSysTableTest.java | 9 +-
21 files changed, 160 insertions(+), 116 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java
index 8e27898b249..a3aa736ed96 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java
@@ -197,8 +197,8 @@ public class IcebergScanRange implements ConnectorScanRange
{
/**
* The table-format-type string BE uses to select its Iceberg reader,
mirroring paimon's
- * {@code "paimon"}: the value of {@code TableFormatType.ICEBERG} (see
fe-core
- * {@code org.apache.doris.datasource.scan.TableFormatType}).
+ * {@code "paimon"}. The connector owns this string: fe-core forwards
whatever it returns to BE, and
+ * holds no per-data-source format constant of its own.
*/
@Override
public String getTableFormatType() {
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java
index 7f451815186..07f52b5d3a5 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java
@@ -58,8 +58,8 @@ public class IcebergScanRangeTest {
Assertions.assertEquals(8192L, range.getFileSize());
Assertions.assertEquals("parquet", range.getFileFormat());
// WHY: BE selects its iceberg reader off
TTableFormatFileDesc.table_format_type, whose value for
- // iceberg is "iceberg" (TableFormatType.ICEBERG). MUTATION: "paimon"
/ "plugin_driven" -> BE routes
- // the split to the wrong reader -> red.
+ // iceberg is "iceberg" -- a string this connector owns, since fe-core
forwards it verbatim.
+ // MUTATION: "paimon" / "plugin_driven" -> BE routes the split to the
wrong reader -> red.
Assertions.assertEquals("iceberg", range.getTableFormatType());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
index a0f80327be2..0fca6040aa0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
@@ -587,13 +587,6 @@ public class Alter {
case ELASTICSEARCH:
processAlterExternalTable(command, (Table) tableIf, (Database)
dbIf);
return;
- case HMS_EXTERNAL_TABLE:
- case JDBC_EXTERNAL_TABLE:
- case ICEBERG_EXTERNAL_TABLE:
- case PAIMON_EXTERNAL_TABLE:
- case MAX_COMPUTE_EXTERNAL_TABLE:
- case HUDI_EXTERNAL_TABLE:
- case TRINO_CONNECTOR_EXTERNAL_TABLE:
case PLUGIN_EXTERNAL_TABLE:
alterOps.addAll(command.getOps());
processAlterTableForExternalTable((ExternalTable) tableIf,
alterOps);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
index ac6d53271f2..e59955c515b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
@@ -58,10 +58,8 @@ import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
-import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
@@ -715,24 +713,10 @@ public class Database extends MetaObject implements
Writable, DatabaseIf<Table>,
@Override
public void write(DataOutput out) throws IOException {
- discardHudiTable();
Text.writeString(out, GsonUtils.GSON.toJson(this));
writeTables(out);
}
-
- private void discardHudiTable() {
- Iterator<Entry<String, Table>> iterator =
nameToTable.entrySet().iterator();
- while (iterator.hasNext()) {
- Map.Entry<String, Table> entry = iterator.next();
- if (entry.getValue().getType() == TableType.HUDI) {
- LOG.warn("hudi table is deprecated, discard it. table name:
{}", entry.getKey());
- iterator.remove();
- idToTable.remove(entry.getValue().getId());
- }
- }
- }
-
public void analyze() {
for (Table table : nameToTable.values()) {
table.analyze(getFullName());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index a98aa7211bb..aee9b67ad61 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -229,8 +229,8 @@ public interface TableIf {
*/
Set<Pair<String, String>> getColumnIndexPairs(Set<String> columns);
- // Get all the chunk sizes of this table. Now, only HMS external table
implemented this interface.
- // For HMS external table, the return result is a list of all the files'
size.
+ // Get all the chunk sizes of this table. Only a plugin-driven external
table whose connector can list
+ // file sizes implements this; the return result is a list of all the
files' size.
List<Long> getChunkSizes();
void write(DataOutput out) throws IOException;
@@ -260,15 +260,25 @@ public interface TableIf {
/**
* Doris table type.
+ *
+ * <p>There is no per-data-source external table type: an external table
served by a connector plugin is a
+ * {@code PLUGIN_EXTERNAL_TABLE}, and the source's own name is answered by
the connector
+ * ({@code PluginDrivenExternalCatalog#getDisplayEngineName}), never by a
mapping held here.</p>
+ *
+ * <p>An image written before the cutover still carries the old per-source
names, and stays readable:
+ * the persisted table class is remapped by {@code GsonUtils}'
compatible-subtype registry, and the stale
+ * {@code type} string deserializes to {@code null} (Gson returns null for
an enum name it does not know,
+ * it does not throw), which {@code
PluginDrivenExternalTable#gsonPostProcess} then normalizes to
+ * {@code PLUGIN_EXTERNAL_TABLE} — the same normalization it already
applied to a recognized legacy name.</p>
+ *
+ * <p>The legacy internal-catalog {@code CREATE EXTERNAL TABLE ...
ENGINE=iceberg|hudi} types are gone as
+ * well: no {@code IcebergTable} / {@code HudiTable} class exists to carry
them, and neither appears in the
+ * 4.1.3 GSON table registry ({@code upgrade/413/labels.tbl.txt}), so no
readable image can hold one.
+ * {@code HIVE} stays because {@code HiveTable} is still registered
there.</p>
*/
enum TableType {
- MYSQL, ODBC, OLAP, SCHEMA, INLINE_VIEW, VIEW, BROKER, ELASTICSEARCH,
HIVE,
- @Deprecated
- ICEBERG, @Deprecated
- HUDI, JDBC,
- TABLE_VALUED_FUNCTION, HMS_EXTERNAL_TABLE, ES_EXTERNAL_TABLE,
MATERIALIZED_VIEW, JDBC_EXTERNAL_TABLE,
- ICEBERG_EXTERNAL_TABLE, TEST_EXTERNAL_TABLE, PAIMON_EXTERNAL_TABLE,
MAX_COMPUTE_EXTERNAL_TABLE,
- HUDI_EXTERNAL_TABLE, TRINO_CONNECTOR_EXTERNAL_TABLE,
LAKESOUl_EXTERNAL_TABLE, DICTIONARY, DORIS_EXTERNAL_TABLE,
+ MYSQL, ODBC, OLAP, SCHEMA, INLINE_VIEW, VIEW, BROKER, ELASTICSEARCH,
HIVE, JDBC,
+ TABLE_VALUED_FUNCTION, MATERIALIZED_VIEW, TEST_EXTERNAL_TABLE,
DICTIONARY, DORIS_EXTERNAL_TABLE,
PLUGIN_EXTERNAL_TABLE,
STREAM;
@@ -290,25 +300,12 @@ public interface TableIf {
return "Broker";
case ELASTICSEARCH:
return "ElasticSearch";
- case ES_EXTERNAL_TABLE:
- return "es";
case HIVE:
return "Hive";
- case HUDI:
- case HUDI_EXTERNAL_TABLE:
- return "Hudi";
case JDBC:
- case JDBC_EXTERNAL_TABLE:
return "jdbc";
case TABLE_VALUED_FUNCTION:
return "Table_Valued_Function";
- case HMS_EXTERNAL_TABLE:
- return "hms";
- case ICEBERG:
- case ICEBERG_EXTERNAL_TABLE:
- return "iceberg";
- case PAIMON_EXTERNAL_TABLE:
- return "paimon";
case DICTIONARY:
return "dictionary";
case DORIS_EXTERNAL_TABLE:
@@ -346,15 +343,9 @@ public interface TableIf {
case BROKER:
case ELASTICSEARCH:
case HIVE:
- case HUDI:
case JDBC:
- case JDBC_EXTERNAL_TABLE:
case TABLE_VALUED_FUNCTION:
- case HMS_EXTERNAL_TABLE:
- case ICEBERG_EXTERNAL_TABLE:
- case PAIMON_EXTERNAL_TABLE:
case MATERIALIZED_VIEW:
- case TRINO_CONNECTOR_EXTERNAL_TABLE:
case DORIS_EXTERNAL_TABLE:
case PLUGIN_EXTERNAL_TABLE:
return "BASE TABLE";
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java
b/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java
index f965bbf0e38..bee844a9349 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java
@@ -439,7 +439,6 @@ public class NereidsSqlCacheManager {
for (Entry<FullTableName, TableVersion> scanTable :
sqlCacheContext.getUsedTables().entrySet()) {
TableVersion tableVersion = scanTable.getValue();
if (tableVersion.type != TableType.OLAP && tableVersion.type !=
TableType.MATERIALIZED_VIEW
- && tableVersion.type != TableType.HMS_EXTERNAL_TABLE
&& tableVersion.type != TableType.PLUGIN_EXTERNAL_TABLE) {
return IsChanged.CHANGED_AND_INVALIDATE_CACHE;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
index 2b1281bdcf0..7b7207790bc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
@@ -33,7 +33,6 @@ import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.infoschema.ExternalInfoSchemaDatabase;
import org.apache.doris.datasource.infoschema.ExternalMysqlDatabase;
-import org.apache.doris.datasource.log.InitDatabaseLog;
import org.apache.doris.datasource.metacache.MetaCache;
import org.apache.doris.datasource.test.TestExternalDatabase;
import org.apache.doris.persist.gson.GsonPostProcessable;
@@ -84,7 +83,6 @@ public abstract class ExternalDatabase<T extends
ExternalTable>
private Map<String, String> lowerCaseToTableName = Maps.newConcurrentMap();
@SerializedName(value = "lastUpdateTime")
protected long lastUpdateTime;
- protected final InitDatabaseLog.Type dbLogType;
protected ExternalCatalog extCatalog;
private MetaCache<T> metaCache;
@@ -99,13 +97,11 @@ public abstract class ExternalDatabase<T extends
ExternalTable>
* @param name Database name.
* @param remoteName Remote database name.
*/
- public ExternalDatabase(ExternalCatalog extCatalog, long id, String name,
String remoteName,
- InitDatabaseLog.Type dbLogType) {
+ public ExternalDatabase(ExternalCatalog extCatalog, long id, String name,
String remoteName) {
this.extCatalog = extCatalog;
this.id = id;
this.name = name;
this.remoteName = remoteName;
- this.dbLogType = dbLogType;
}
public void setExtCatalog(ExternalCatalog extCatalog) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/doris/RemoteDorisExternalDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/doris/RemoteDorisExternalDatabase.java
index bd28a140807..bebaf752091 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/doris/RemoteDorisExternalDatabase.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/doris/RemoteDorisExternalDatabase.java
@@ -19,11 +19,10 @@ package org.apache.doris.datasource.doris;
import org.apache.doris.datasource.ExternalCatalog;
import org.apache.doris.datasource.ExternalDatabase;
-import org.apache.doris.datasource.log.InitDatabaseLog;
public class RemoteDorisExternalDatabase extends
ExternalDatabase<RemoteDorisExternalTable> {
public RemoteDorisExternalDatabase(ExternalCatalog extCatalog, long id,
String name, String remoteName) {
- super(extCatalog, id, name, remoteName,
InitDatabaseLog.Type.REMOTE_DORIS);
+ super(extCatalog, id, name, remoteName);
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalInfoSchemaDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalInfoSchemaDatabase.java
index 05425dc6eaa..bef82c0fe28 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalInfoSchemaDatabase.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalInfoSchemaDatabase.java
@@ -22,7 +22,6 @@ import org.apache.doris.catalog.SchemaTable;
import org.apache.doris.datasource.ExternalCatalog;
import org.apache.doris.datasource.ExternalDatabase;
import org.apache.doris.datasource.ExternalTable;
-import org.apache.doris.datasource.log.InitDatabaseLog.Type;
import com.google.common.collect.Lists;
@@ -36,7 +35,7 @@ public class ExternalInfoSchemaDatabase extends
ExternalDatabase {
* @param dbId The id of this database.
*/
public ExternalInfoSchemaDatabase(ExternalCatalog extCatalog, long dbId) {
- super(extCatalog, dbId, InfoSchemaDb.DATABASE_NAME,
InfoSchemaDb.DATABASE_NAME, Type.INFO_SCHEMA_DB);
+ super(extCatalog, dbId, InfoSchemaDb.DATABASE_NAME,
InfoSchemaDb.DATABASE_NAME);
}
public static List<String> listTableNames() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalMysqlDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalMysqlDatabase.java
index 20ef7c65149..a0f84696884 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalMysqlDatabase.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/infoschema/ExternalMysqlDatabase.java
@@ -22,7 +22,6 @@ import org.apache.doris.catalog.MysqlDb;
import org.apache.doris.datasource.ExternalCatalog;
import org.apache.doris.datasource.ExternalDatabase;
import org.apache.doris.datasource.ExternalTable;
-import org.apache.doris.datasource.log.InitDatabaseLog.Type;
import com.google.common.collect.Lists;
@@ -36,7 +35,7 @@ public class ExternalMysqlDatabase extends ExternalDatabase {
* @param dbId The id of this database.
*/
public ExternalMysqlDatabase(ExternalCatalog extCatalog, long dbId) {
- super(extCatalog, dbId, MysqlDb.DATABASE_NAME, MysqlDb.DATABASE_NAME,
Type.INFO_SCHEMA_DB);
+ super(extCatalog, dbId, MysqlDb.DATABASE_NAME, MysqlDb.DATABASE_NAME);
}
public static List<String> listTableNames() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitCatalogLog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitCatalogLog.java
index c7154c6413d..f97d5cd04c4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitCatalogLog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitCatalogLog.java
@@ -32,6 +32,17 @@ import java.util.List;
@Data
public class InitCatalogLog implements Writable {
+ /**
+ * Also the type of the persisted {@code ExternalCatalog#logType} field,
which is why the per-data-source
+ * values stay even though only PLUGIN / REMOTE_DORIS / TEST are still
produced: an image written before
+ * the connector cutover carries one of the old names, and
+ * {@code PluginDrivenExternalCatalog#gsonPostProcess} reads it to
backfill the catalog's {@code type}
+ * property for the resource-backed catalogs (es, jdbc) that never
persisted one. Deleting a name would
+ * make it deserialize to null and lose that catalog's type on upgrade.
+ *
+ * <p>HUDI is absent because it was never produced: hudi tables have
always lived in an hms catalog and
+ * no {@code HudiExternalCatalog} ever existed, so no image can carry
it.</p>
+ */
public enum Type {
HMS,
ES,
@@ -39,7 +50,6 @@ public class InitCatalogLog implements Writable {
ICEBERG,
PAIMON,
MAX_COMPUTE,
- HUDI,
LAKESOUL,
TEST,
TRINO_CONNECTOR,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitDatabaseLog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitDatabaseLog.java
index 1bfb4b7507b..a96edaff3c8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitDatabaseLog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/log/InitDatabaseLog.java
@@ -30,25 +30,16 @@ import java.io.DataOutput;
import java.io.IOException;
import java.util.List;
+/**
+ * Deserialize-only journal payload: {@code OP_INIT_EXTERNAL_DB} has been
ignored on replay since 4.0 and
+ * nothing writes one any more ({@code EditLog#logInitExternalDb} has no
callers). The class stays so that
+ * {@code JournalEntity} can still consume such an entry out of an old
journal; its contents are discarded.
+ *
+ * <p>The former per-data-source {@code Type} enum is gone with it: its only
other user was
+ * {@code ExternalDatabase#dbLogType}, a field that was assigned and never
read.</p>
+ */
@Data
public class InitDatabaseLog implements Writable {
- public enum Type {
- HMS,
- ICEBERG,
- ES,
- JDBC,
- MAX_COMPUTE,
- HUDI,
- PAIMON,
- LAKESOUL,
- TEST,
- INFO_SCHEMA_DB,
- TRINO_CONNECTOR,
- REMOTE_DORIS,
- PLUGIN,
- UNKNOWN;
- }
-
@SerializedName(value = "catalogId")
private long catalogId;
@@ -76,9 +67,6 @@ public class InitDatabaseLog implements Writable {
@SerializedName(value = "remoteTableNames")
private List<String> remoteTableNames;
- @SerializedName(value = "type")
- private Type type;
-
@SerializedName(value = "lastUpdateTime")
protected long lastUpdateTime;
@@ -92,7 +80,6 @@ public class InitDatabaseLog implements Writable {
createTableIds = Lists.newArrayList();
createTableNames = Lists.newArrayList();
remoteTableNames = Lists.newArrayList();
- type = Type.UNKNOWN;
}
public void addRefreshTable(long id, String remoteName) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java
index 8d32afa0187..761fffad76a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java
@@ -24,7 +24,6 @@ import org.apache.doris.connector.api.ConnectorMetadata;
import org.apache.doris.connector.api.ConnectorSession;
import org.apache.doris.datasource.ExternalCatalog;
import org.apache.doris.datasource.ExternalDatabase;
-import org.apache.doris.datasource.log.InitDatabaseLog;
import org.apache.doris.datasource.mvcc.PluginDrivenMvccExternalTable;
/**
@@ -37,12 +36,12 @@ public class PluginDrivenExternalDatabase extends
ExternalDatabase<PluginDrivenE
/** No-arg constructor for GSON deserialization. */
public PluginDrivenExternalDatabase() {
- super(null, 0, null, null, InitDatabaseLog.Type.PLUGIN);
+ super(null, 0, null, null);
}
public PluginDrivenExternalDatabase(ExternalCatalog extCatalog, long id,
String name, String remoteName) {
- super(extCatalog, id, name, remoteName, InitDatabaseLog.Type.PLUGIN);
+ super(extCatalog, id, name, remoteName);
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/TableFormatType.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/TableFormatType.java
index f9bc9c364fc..9cb90b4bec0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/TableFormatType.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/TableFormatType.java
@@ -17,15 +17,14 @@
package org.apache.doris.datasource.scan;
+/**
+ * The {@code TTableFormatFileDesc.table_format_type} values produced by
fe-core itself.
+ *
+ * <p>A connector plugin names its own format string through {@code
ConnectorScanRange#getTableFormatType()},
+ * which {@code PluginDrivenScanNode} forwards to BE verbatim, so no
per-data-source constant belongs here.
+ * This enum is not persisted anywhere — it only builds the thrift string.</p>
+ */
public enum TableFormatType {
- HIVE("hive"),
- ICEBERG("iceberg"),
- HUDI("hudi"),
- PAIMON("paimon"),
- MAX_COMPUTE("max_compute"),
- TRANSACTIONAL_HIVE("transactional_hive"),
- LAKESOUL("lakesoul"),
- TRINO_CONNECTOR("trino_connector"),
TVF("tvf"),
REMOTE_DORIS("remote_doris");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/test/TestExternalDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/test/TestExternalDatabase.java
index d30f353c999..52e27296397 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/test/TestExternalDatabase.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/test/TestExternalDatabase.java
@@ -19,12 +19,11 @@ package org.apache.doris.datasource.test;
import org.apache.doris.datasource.ExternalCatalog;
import org.apache.doris.datasource.ExternalDatabase;
-import org.apache.doris.datasource.log.InitDatabaseLog;
public class TestExternalDatabase extends ExternalDatabase<TestExternalTable> {
public TestExternalDatabase(ExternalCatalog extCatalog, long id, String
name, String remoteName) {
- super(extCatalog, id, name, remoteName, InitDatabaseLog.Type.TEST);
+ super(extCatalog, id, name, remoteName);
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
index 73a5bfd844a..53949892279 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
@@ -764,10 +764,6 @@ public class BindRelation extends OneAnalysisRuleFactory {
Plan viewBody = parseAndAnalyzeDorisView(view,
qualifiedTableName, cascadesContext);
LogicalView<Plan> logicalView = new LogicalView<>(view,
viewBody);
return new LogicalSubQueryAlias<>(qualifiedTableName,
logicalView);
- case PAIMON_EXTERNAL_TABLE:
- case MAX_COMPUTE_EXTERNAL_TABLE:
- case TRINO_CONNECTOR_EXTERNAL_TABLE:
- case LAKESOUl_EXTERNAL_TABLE:
case PLUGIN_EXTERNAL_TABLE:
if (table instanceof PluginDrivenExternalTable
&& ((PluginDrivenExternalTable) table).isView()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java
index 56c565e4c01..8107828fa92 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java
@@ -241,9 +241,7 @@ public class ShowPartitionsCommand extends ShowCommand {
}
DatabaseIf db = catalog.getDbOrAnalysisException(dbName);
- TableIf table = db.getTableOrMetaException(tblName, TableType.OLAP,
- TableType.HMS_EXTERNAL_TABLE,
TableType.MAX_COMPUTE_EXTERNAL_TABLE, TableType.PAIMON_EXTERNAL_TABLE,
- TableType.PLUGIN_EXTERNAL_TABLE);
+ TableIf table = db.getTableOrMetaException(tblName, TableType.OLAP,
TableType.PLUGIN_EXTERNAL_TABLE);
if (!catalog.isInternalCatalog()) {
if (!table.isPartitionedTable()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java
index 2a53861fd3a..1a3c8045452 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java
@@ -124,7 +124,6 @@ public class PartitionValuesTableValuedFunction extends
MetadataTableValuedFunct
TableIf table;
try {
table = db.get().getTableOrMetaException(tableName, TableType.OLAP,
- TableType.HMS_EXTERNAL_TABLE,
TableType.MAX_COMPUTE_EXTERNAL_TABLE,
TableType.PLUGIN_EXTERNAL_TABLE);
} catch (MetaNotFoundException e) {
throw new AnalysisException(e.getMessage(), e);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java
index e37575a302e..b58f2bed9ab 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java
@@ -178,7 +178,6 @@ public class PartitionsTableValuedFunction extends
MetadataTableValuedFunction {
TableIf table = null;
try {
table = db.get().getTableOrMetaException(tableName, TableType.OLAP,
- TableType.HMS_EXTERNAL_TABLE,
TableType.MAX_COMPUTE_EXTERNAL_TABLE,
TableType.PLUGIN_EXTERNAL_TABLE);
} catch (MetaNotFoundException e) {
throw new AnalysisException(e.getMessage(), e);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/LegacyExternalTableTypeReplayTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/LegacyExternalTableTypeReplayTest.java
new file mode 100644
index 00000000000..8d005504592
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/LegacyExternalTableTypeReplayTest.java
@@ -0,0 +1,99 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource;
+
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.catalog.TableIf.TableType;
+import org.apache.doris.datasource.mvcc.PluginDrivenMvccExternalTable;
+import org.apache.doris.datasource.plugin.PluginDrivenExternalTable;
+import org.apache.doris.persist.gson.GsonUtils;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Guards the OTHER half of edit-log compatibility for the SPI cutover. The
{@code *GsonCompatReplayTest}
+ * family covers the persisted {@code "clazz"} discriminator; this covers the
persisted {@code "type"} value,
+ * which is a {@link TableType} enum name serialized as a bare string.
+ *
+ * <p><b>Why this test exists:</b> the per-data-source {@code TableType}
constants (HMS_EXTERNAL_TABLE,
+ * ICEBERG_EXTERNAL_TABLE, ...) were deleted so that fe-core holds no data
source names, but an image written
+ * by a pre-cutover FE still contains those strings. Deleting a persisted enum
constant is only safe because
+ * of a two-step property that nothing in the compiler enforces: GSON returns
{@code null} (rather than
+ * throwing) for an enum name it does not know, and {@link
PluginDrivenExternalTable#gsonPostProcess()} then
+ * normalizes whatever it got to {@code PLUGIN_EXTERNAL_TABLE}. If either step
regresses, the failure is not
+ * a compile error but an FE that replays a persisted external table with a
{@code null} type and NPEs (or
+ * silently misroutes it) on the next query — so it must be pinned by a test.
+ *
+ * <p>MUTATION: dropping the type normalization in {@code gsonPostProcess}
leaves the restored type
+ * {@code null} -> red for every legacy name below.</p>
+ */
+public class LegacyExternalTableTypeReplayTest {
+
+ /**
+ * Every per-source table type a pre-cutover FE could have written into an
external table's {@code type}
+ * field, and which this branch no longer declares. Deliberately spelled
as string literals: these names
+ * must NOT come back as constants just to keep a test compiling.
+ */
+ private static final String[] DELETED_LEGACY_TYPES = {
+ "HMS_EXTERNAL_TABLE",
+ "ES_EXTERNAL_TABLE",
+ "JDBC_EXTERNAL_TABLE",
+ "ICEBERG_EXTERNAL_TABLE",
+ "PAIMON_EXTERNAL_TABLE",
+ "MAX_COMPUTE_EXTERNAL_TABLE",
+ "HUDI_EXTERNAL_TABLE",
+ "TRINO_CONNECTOR_EXTERNAL_TABLE",
+ "LAKESOUl_EXTERNAL_TABLE",
+ };
+
+ @Test
+ public void deletedLegacyTypeNamesReplayAsPluginExternalTable() {
+ for (String legacyType : DELETED_LEGACY_TYPES) {
+ // Both targets of the compatible-subtype registry: the base class
(es/jdbc/maxcompute/trino/
+ // lakesoul migrate here) and the MVCC variant (hms/iceberg/paimon
migrate here).
+ assertReplaysAsPluginType(new PluginDrivenExternalTable(),
legacyType);
+ assertReplaysAsPluginType(new PluginDrivenMvccExternalTable(),
legacyType);
+ }
+ }
+
+ /**
+ * Round-trips a valid plugin-driven table through GSON, rewrites only the
persisted {@code "type"} value
+ * to a legacy name (faithfully reproducing old-image bytes without
depending on the deleted constants),
+ * then asserts the replay both survives and lands on {@code
PLUGIN_EXTERNAL_TABLE}.
+ */
+ private void assertReplaysAsPluginType(PluginDrivenExternalTable table,
String legacyType) {
+ table.id = 7L;
+ table.name = "legacy_tbl";
+ table.dbName = "legacy_db";
+ table.type = TableType.PLUGIN_EXTERNAL_TABLE;
+
+ String needle = "\"type\":\"PLUGIN_EXTERNAL_TABLE\"";
+ String json = GsonUtils.GSON.toJson(table, TableIf.class);
+ // Sanity: the type must actually be persisted, or rewriting it below
would prove nothing.
+ Assertions.assertTrue(json.contains(needle),
+ "expected " + needle + " in serialized json: " + json);
+ String legacyJson = json.replace(needle, "\"type\":\"" + legacyType +
"\"");
+
+ TableIf restored = GsonUtils.GSON.fromJson(legacyJson, TableIf.class);
+ Assertions.assertSame(table.getClass(), restored.getClass(),
+ "a legacy '" + legacyType + "' type value must not change
which class replays");
+ Assertions.assertEquals(TableType.PLUGIN_EXTERNAL_TABLE,
restored.getType(),
+ "a table persisted as '" + legacyType + "' must replay as
PLUGIN_EXTERNAL_TABLE, not null");
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java
index 92a1b1c5526..31bbdcc55a0 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java
@@ -290,12 +290,11 @@ public class PluginDrivenSysTableTest {
PluginDrivenSysExternalTable sys = new
PluginDrivenSysExternalTable(base, "snapshots");
// WHY: information_schema.tables.TABLE_TYPE for an iceberg sys table
(e.g. tbl$snapshots) must read
- // "BASE TABLE", byte-identical to a legacy ICEBERG_EXTERNAL_TABLE.
The sys table inherits
- // PLUGIN_EXTERNAL_TABLE and routes getMysqlType ->
TableType.toMysqlType; the same-test pin of
- // ICEBERG_EXTERNAL_TABLE.toMysqlType proves new == legacy with no
Env. MUTATION: deleting the
+ // "BASE TABLE", byte-identical to what a legacy
ICEBERG_EXTERNAL_TABLE reported. The sys table inherits
+ // PLUGIN_EXTERNAL_TABLE and routes getMysqlType ->
TableType.toMysqlType. The legacy type constant is
+ // gone, so the legacy answer is pinned here as the frozen literal it
always was. MUTATION: deleting the
// PLUGIN_EXTERNAL_TABLE case in TableIf.TableType.toMysqlType -> sys
getMysqlType returns null -> red.
- Assertions.assertEquals("BASE TABLE", sys.getMysqlType());
- Assertions.assertEquals("BASE TABLE",
TableType.ICEBERG_EXTERNAL_TABLE.toMysqlType(),
+ Assertions.assertEquals("BASE TABLE", sys.getMysqlType(),
"the new plugin sys path must match the legacy iceberg
TABLE_TYPE");
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
