JackieTien97 commented on code in PR #19237:
URL: https://github.com/apache/iotdb/pull/19237#discussion_r4561915369
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/parser/StatementGenerator.java:
##########
@@ -385,6 +389,19 @@ public static InsertTabletStatement
createStatement(TSInsertTabletReq insertTabl
ColumnCategory.values()[insertTabletReq.columnCategories.get(i).intValue()]);
}
insertStatement.setColumnCategories(columnCategories);
+ if (insertTabletReq.isSetDatabase()) {
+ if (insertTabletReq.getDatabase().isEmpty()) {
+ throw new IllegalArgumentException(
Review Comment:
Malformed names come back with inconsistent status codes:
| `database` | result |
|---|---|
| `""` | 301 `[EXECUTE_STATEMENT_ERROR(301)] Exception occurred:
insertTablet failed. database is not specified for insert:` |
| `" "` | 500 `Unknown database ` |
| `"a.b"` | 500 `Unknown database a.b` |
| `null` with no session database | 701 `database is not specified` |
The DataNode also logs each 301/500 response as a WARN with a full stack
trace, on every call. For `""` the cause is that
`ErrorHandlingUtils.onQueryException` doesn't recognize
`IllegalArgumentException`, so the exception falls through to
`onNpeOrUnexpectedException` (DN log: `StatementGenerator.java:394`). In
addition, `DATABASE_IS_NOT_SPECIFIED_FOR_INSERT` is a prefix that is supposed
to be followed by the statement, as in the relational `Analyzer`. Used here,
the message ends in a dangling colon (zh: "未指定插入操作的数据库:").
`CREATE DATABASE` already validates names with
`TableConfigTaskVisitor.validateDatabaseName`. It rejects `""`, blanks, `.`,
characters outside `DATABASE_PATTERN`, and names longer than 64. It throws a
`SemanticException` that wraps `IllegalPathException`, which comes back as 509
and is logged at INFO without a stack trace. Calling it after lower-casing (see
the comment above) gives every malformed name a clean user error.
##########
iotdb-protocol/thrift-datanode/src/main/thrift/client.thrift:
##########
@@ -274,6 +274,7 @@ struct TSInsertTabletReq {
11: optional bool isCompressed
12: optional list<byte> encodingTypes
13: optional byte compressType
+ 14: optional string database
Review Comment:
**Blocking: an older server ignores this field and writes into the session
database.**
Thrift servers skip unknown optional fields, so a server without this change
never sees `database` and falls back to the session's current database. I ran
the PR client against a server built from 207503295e:
```java
// session opened with database("cdb1"); cdb1.t1 and cdb2.t1 both exist
session.insert(tablet("t1", 3 rows), "cdb2"); // returns OK
session.insert(tablet("only_in_cdb2", 2 rows), "cdb2"); // returns OK
```
Result: `cdb1.t1` gets the 3 rows, `cdb2.t1` stays at 0, and table
`only_in_cdb2` is auto-created in `cdb1`. With no session database the call
fails with 701, but whenever a session database is set the misrouting is silent.
The description acknowledges this fallback ("an older server can ignore the
unknown optional field and fall back to its session database"). Documenting it
isn't enough, though: the call reports success while the data lands in a
different database, and the caller has no way to detect that. A rolling upgrade
is a realistic way to hit this, since a redirect can send the request to a node
that hasn't been upgraded yet.
Suggestion: in `openSession`, have the server advertise a capability in
`TSOpenSessionResp.configuration`, next to the existing time-precision entry.
The client records it per `SessionConnection`, because redirect connections are
opened separately. When `database != null` and the connection lacks the
capability, the client throws before it sends the request.
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/session/IoTDBSessionRelationalIT.java:
##########
@@ -177,6 +177,63 @@ public static void main(String[] args)
}
}
+ @Test
+ public void testInsertTabletIntoSpecifiedDatabase()
+ throws IoTDBConnectionException, StatementExecutionException {
+ try (ITableSession session =
EnvFactory.getEnv().getTableSessionConnection()) {
+ session.executeNonQueryStatement(
+ "CREATE TABLE db1.tablet_database (tag1 STRING TAG, m1 DOUBLE
FIELD)");
+ session.executeNonQueryStatement(
+ "CREATE TABLE db2.tablet_database (tag1 STRING TAG, m1 DOUBLE
FIELD)");
+
+ List<String> columnNames = Arrays.asList("tag1", "m1");
+ List<TSDataType> dataTypes = Arrays.asList(TSDataType.STRING,
TSDataType.DOUBLE);
+ List<ColumnCategory> columnCategories =
+ Arrays.asList(ColumnCategory.TAG, ColumnCategory.FIELD);
+ Tablet tablet = new Tablet("tablet_database", columnNames, dataTypes,
columnCategories, 2);
+ for (int row = 0; row < 2; row++) {
+ tablet.addTimestamp(row, row);
+ tablet.addValue("tag1", row, "tag:" + row);
+ tablet.addValue("m1", row, row * 1.0);
+ }
+
+ session.insert(tablet, "db1");
+ session.executeNonQueryStatement("USE db1");
+ session.insert(tablet, "db2");
+ Tablet currentDatabaseTablet =
+ new Tablet("tablet_database", columnNames, dataTypes,
columnCategories, 1);
+ currentDatabaseTablet.addTimestamp(0, 2);
+ currentDatabaseTablet.addValue("tag1", 0, "tag:2");
+ currentDatabaseTablet.addValue("m1", 0, 2.0);
+ session.insert(currentDatabaseTablet, null);
+
+ try (SessionDataSet dataSet =
+ session.executeQueryStatement("SELECT * FROM db1.tablet_database")) {
+ assertTrue(dataSet.hasNext());
+ dataSet.next();
+ assertTrue(dataSet.hasNext());
+ dataSet.next();
+ assertTrue(dataSet.hasNext());
+ dataSet.next();
+ assertFalse(dataSet.hasNext());
+ }
+ try (SessionDataSet dataSet =
+ session.executeQueryStatement("SELECT * FROM db2.tablet_database")) {
+ assertTrue(dataSet.hasNext());
+ dataSet.next();
+ assertTrue(dataSet.hasNext());
+ dataSet.next();
+ assertFalse(dataSet.hasNext());
+ }
+ try (SessionDataSet dataSet = session.executeQueryStatement("SHOW
CURRENT_DATABASE")) {
+ assertTrue(dataSet.hasNext());
+ assertEquals("db1", dataSet.next().getField(0).getStringValue());
+ }
+
+ assertThrows(StatementExecutionException.class, () ->
session.insert(tablet, ""));
Review Comment:
Please assert the status code here, not just the exception type. `""`
currently returns 301, the generic failure code, so this assertion would still
pass if the call failed for an unrelated reason.
Missing cases:
- **Tree-prefixed names.** Try `"root.xxx"`, `"ROOT.xxx"` and `"Root.xxx"`
while both tree database `root.xxx` and table database `xxx` exist. Assert the
error code, and assert that no device appears under `root.xxx.**` (see the
comment on `StatementGenerator`).
- **The grouped path with an explicit database.** No test runs
`insertRelationalTabletByGroup` with a database set. The new unit test uses a
single device. In the IT, each explicit-database insert starts with an empty
route map for that database, so it always goes to the default connection.
Extending `SessionCacheLeaderTest` with a multi-device tablet whose devices
have different cached leaders, written to an explicit database that differs
from the session's, would cover the grouping, the redirect bookkeeping and the
per-group requests carrying the database.
##########
iotdb-client/isession/src/main/java/org/apache/iotdb/isession/ISession.java:
##########
@@ -70,7 +70,15 @@ void open(
boolean enableRPCCompression,
int connectionTimeoutInMs,
Map<String, TEndPoint> deviceIdToEndpoint,
- Map<IDeviceID, TEndPoint> tabletModelDeviceIdToEndpoint,
+ Map<IDeviceID, TEndPoint> tableModelDeviceIdToEndpoint,
+ INodeSupplier nodeSupplier)
+ throws IoTDBConnectionException;
+
+ void openWithTableModelDatabaseCache(
Review Comment:
This adds an internal cache-sharing hook to `ISession`, a public interface
that users program against, and its nested-map parameter exposes the cache
layout. The only callers are in `SessionPool`. The method has to be on the
interface only because `getSession()` declares `ISession session =
queue.poll();`. Opening the new connection through a local `Session` would
avoid that:
```java
Session newSession = constructNewSession();
newSession.openWithTableModelDatabaseCache(...);
session = newSession;
```
`tryConstructNewSession()` already uses a `Session`.
Separately, the new abstract methods here and the new
`ITableSession.insert(Tablet, String)` break every third-party implementation
of these interfaces at compile time. `ISession` already uses `default` methods
for this kind of evolution (e.g. `isEnableCacheLeader`). `insert(Tablet,
String)` could be a `default` method that delegates to `insert(tablet)` when
`database == null` and throws `UnsupportedOperationException` otherwise.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/parser/StatementGenerator.java:
##########
@@ -385,6 +389,19 @@ public static InsertTabletStatement
createStatement(TSInsertTabletReq insertTabl
ColumnCategory.values()[insertTabletReq.columnCategories.get(i).intValue()]);
}
insertStatement.setColumnCategories(columnCategories);
+ if (insertTabletReq.isSetDatabase()) {
+ if (insertTabletReq.getDatabase().isEmpty()) {
+ throw new IllegalArgumentException(
+ DataNodeQueryMessages.DATABASE_IS_NOT_SPECIFIED_FOR_INSERT);
+ }
+ if (!PathUtils.isTableModelDatabase(insertTabletReq.getDatabase())) {
Review Comment:
**Blocking: the tree-model check is case-sensitive and runs before
lower-casing, so `ROOT.xxx` gets through and the rows are written into the tree
database.**
`PathUtils.isTableModelDatabase` is just `!startsWith("root.")`, so
`"ROOT.treedb"` and `"Root.treedb"` pass. The name is then lower-cased to
`root.treedb`. Schema validation and the privilege check resolve the table
through `unQualifyDatabaseName(...)`, which gives table `treedb.tt`, while
schema/data partitioning and device creation use `root.treedb`. The other entry
points don't have this gap: `openSession` lower-cases before
`IClientSession.setDatabaseName` checks the name, and SQL rejects a
`"ROOT.xxx".t` prefix with 701 (`The tree model database shall not be specified
in table model.`).
Repro on this head. Tree database `root.treedb` has device `d1`, and table
database `treedb` has table `tt(tag1 TAG, m1 FIELD)`:
```java
session.insert(tablet("tt", tags d0, d1; 2 rows), "ROOT.treedb"); // OK
session.insert(tablet("tt", 1 row), "Root.treedb"); // OK
session.insert(tablet("tt", 1 row), "treedb"); // OK
(control)
```
| | before | after |
|---|---|---|
| table `SELECT count(*) FROM treedb.tt` | 0 | 1 (only the control row) |
| tree `COUNT DEVICES root.treedb.**` | 1 | 3: new aligned devices
`root.treedb.tt.d0` and `root.treedb.tt.d1` |
| tree `COUNT TIMESERIES root.treedb.**` | 1 | 1 |
The rows are written: after `FLUSH` they are in a TsFile under
`sequence/root.treedb/...`. Neither model can read them, though. Tree `SELECT *
FROM root.treedb.tt.d0` returns 0 rows and `SHOW TIMESERIES root.treedb.tt.**`
is empty.
This also gets around the tree privilege. Take a user that has only `GRANT
INSERT ON DATABASE treedb`:
- tree `INSERT INTO root.treedb.d9(time, s1) VALUES (1, 1)` → 803, needs
`WRITE_DATA on [root.treedb.d9.s1]`
- `insert(tablet, "root.treedb")` → 701
- `insert(tablet, "ROOT.treedb")` → **OK**, and device `root.treedb.tt.pv2`
is created in the tree database
If there is no matching table database, the same input surfaces internal
errors instead of a clean 701:
- `"ROOT.db1"` → 500 `Create SchemaPartition failed because the database:
root.db1 is not exists`
- `"ROOT.db1"` with a table that doesn't exist yet → 301
`org.apache.iotdb.confignode.exception.DatabaseNotExistsException: Database:
root.db1 doesn't exist.`
- `"ROOT.treedb"` before table database `treedb` existed → 500 `Unknown
database treedb`
Also, the reused `DATABASE_NAME_IS_TREE_MODEL_DATABASE` ends with "...which
is not allowed to set in the client session." (zh: "...不允许设置到客户端会话中。"). This
API deliberately leaves the session untouched.
`THE_TREE_MODEL_DATABASE_SHALL_NOT_BE_SPECIFIED`, which the SQL path uses for
the same case, fits better.
Suggested shape:
```java
final String database =
insertTabletReq.getDatabase().toLowerCase(Locale.ENGLISH);
if (!PathUtils.isTableModelDatabase(database)) {
throw new SemanticException(
DataNodeQueryMessages.THE_TREE_MODEL_DATABASE_SHALL_NOT_BE_SPECIFIED);
}
TableConfigTaskVisitor.validateDatabaseName(database); // see the comment on
the empty-name check
insertStatement.setDatabaseName(database);
```
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/Session.java:
##########
@@ -3101,6 +3166,12 @@ private void
insertRelationalTabletByGroup(Map<SessionConnection, Tablet> relati
}
}
+ private static void setRelationalRequestDatabase(TSInsertTabletReq request,
String database) {
Review Comment:
Nit: the null check is redundant. The generated `setDatabase(null)` leaves
the field unset, because `isSetDatabase()` is `database != null`. More
importantly, the database is attached at three separate call sites, each right
after `setWriteToTable(true)`, so a future send path can easily forget it. A
single builder for relational tablet requests, taking the tablet and the
database, would keep this in one place. That builder is also the natural home
for the server-capability check suggested on `client.thrift`.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/Session.java:
##########
@@ -1524,11 +1556,16 @@ private void handleRedirection(IDeviceID deviceId,
TEndPoint endpoint) {
}
});
if (connection == null) {
- tableModelDeviceIdToEndpoint.remove(deviceId);
+ databaseToEndpoint.remove(deviceId, endpoint);
}
}
}
+ private String resolveTableModelRouteDatabase(String database) {
+ String routeDatabase = database == null ? this.database : database;
Review Comment:
Nit: the route-cache key is the raw string, while the server lower-cases it,
so `"DB1"` and `"db1"` get separate route maps for the same database.
`getTableModelDatabaseRoutes` also calls `computeIfAbsent` before the server
has validated the name. Every invalid name (`""`, `"ROOT.x"`, typos) therefore
leaves an empty entry in a map that a `SessionPool` shares across sessions.
Lower-casing here, as the server does, and only creating the entry when a
redirect is actually recorded would avoid both problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]