yuqi1129 commented on code in PR #11033:
URL: https://github.com/apache/gravitino/pull/11033#discussion_r3231416233
##########
core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/OwnerMetaBaseSQLProvider.java:
##########
@@ -260,13 +260,18 @@ public String selectOwnerByMetadataObjectIdAndType(
+ " ORDER BY updated_at DESC, id DESC LIMIT 1";
}
- public String selectChangedOwners(@Param("updatedAtFrom") long
updatedAtFrom) {
- return "SELECT metadata_object_id as metadataObjectId,"
+ public String selectChangedOwners(@Param("lastId") long lastId) {
+ return "SELECT id,"
+ + " metadata_object_id as metadataObjectId,"
+ " metadata_object_type as metadataObjectType,"
+ " updated_at as updatedAt"
+ " FROM "
+ OWNER_TABLE_NAME
- + " WHERE deleted_at = 0 AND updated_at >= #{updatedAtFrom}"
- + " ORDER BY updated_at, id LIMIT 1000";
+ + " WHERE deleted_at = 0 AND id > #{lastId}"
+ + " ORDER BY id LIMIT 1000";
+ }
+
+ public String selectLatestChangeId() {
+ return "SELECT COALESCE(MAX(id), 0) FROM " + OWNER_TABLE_NAME + " WHERE
deleted_at = 0";
}
Review Comment:
> How do we maintain the last id? For example, if we restart the server,
will the last id start from 0, can you explain more?
Besides, is it "last" or "latest"? You should be consistent in everywhere.
`Id` is the primary key and is auto-increment, it will be managed by the
database, and has nothing to do with the Gravitno server. in the point, `last`
and `latest` has the same syntax. anyway, Let me make them consistent.
Will we have an issue that if we don't mark the deleted_at = 0 in time, will
we possibly read the entity log twice? Does it have side effects?
It could not be possible.
1. Gravitino will fetch the largest(latest) ID and start fetching the data
when Gravitino starts.
2. A background thread will mark the change log as deleted when the log is
out-of-date. For example: we can freely remove all logs that happened about 1
hour ago.
3. Repeatedly fetching data is not a problem, as we only accept the `drop`
and `alter` entity change log, and then extract the name, evict it from the
cache. It's idempotent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
