yuqi1129 opened a new issue, #13853: URL: https://github.com/apache/gravitino/issues/13853
### Describe the proposal We stress-tested the 2.0.0 EntityStore OCC on `main` (`46cbef5bdd`) and compared it with the commit before OCC (`550f701751`, the parent of #12454). Multiple threads call `RelationalBackend` directly to simulate several servers sharing one database. Every test ran on H2, MySQL 8.0 and PostgreSQL 13. Scope: the entity store itself, meaning `EntityStore`, `RelationalBackend`, `JDBCBackend`, the meta services and the mappers. Consistency between the store and external catalogs (Hive, Iceberg, Lance, ...) is out of scope. **What OCC already gets right.** The scenarios below fail on the pre-OCC commit and pass on `main`, on all three backends: - No lost updates under concurrent alters of a table, schema, catalog, fileset, topic, view or function. - No orphan tables or columns when tables are created during a cascading schema drop. - No dangling version, column, tag, owner, policy-tag, user-role or securable-object rows when a drop races an update. - When two renames and a create race for the same name, exactly one wins. - Concurrent model-version links get distinct version numbers. - Concurrent overwrites with the same id stay consistent. **Problems found** | # | Problem | Evidence | |---|---|---| | 1 | A table read returns **0 columns** if the table is overwritten or dropped between the read of the table row and the read of its columns. During an overwrite the table exists the whole time, yet readers still get 0 columns. A torn read may also be put into the entity cache. #13181 described this problem and was closed as not planned. | `TableMetaService.getTableByIdentifier` reads the table row and the columns in two statements. Out of all reads, the number that came back with 0 columns: MySQL 72/823, PostgreSQL 44/563, H2 6/2425. | | 2 | A cascading catalog or metalake drop soft-deletes all schemas with one `(schema_id=? AND current_version=?) OR ...` statement, and its cost grows superlinearly with the number of schemas. The catalog/metalake and schema row locks are held for the whole statement. | 17,000 schemas on MySQL: 24.6 s, versus 162 ms before OCC. The single UPDATE scans only 17,000 rows, so the time goes to the optimizer. PostgreSQL 1.1 s vs 0.11 s; H2 4.7 s vs 0.66 s. `SchemaMetaBaseSQLProvider#softDeleteSchemaMetasWithVersion` and the PG/catalog variants. All callers go through `OccWriteSupport#deleteChildrenWithVersions`. | | 3 | Model-version get and list fail with `OptimisticLockException` (409) while other versions of the same model are being linked. `readWithStableModel` makes 3 attempts with no backoff, and every link advances the model's aggregate version. MODEL_VERSION is not cacheable, so these failures reach users. | Get of an unrelated alias with 4 concurrent linkers fails 22–35% of the time; list fails 56–94%. The comment in the code points to #13181, which is closed. | | 4 | When `force=false`, `MetalakeManager#dropMetalake` lists catalogs, finds none, and then calls `store.delete(ident, METALAKE, true)`. A catalog created by another server in between is silently cascade-deleted. The store's non-cascade path already performs this emptiness check atomically under the metalake row lock. | Found by reading the code. | | 5 | The default `deleteAndGet` (get, then delete by name) is used for every type except FILESET. With a rename and a re-create in between, it returns entity A but deletes the new entity B. | A deterministic interleaving reproduces it on all three backends. This is already tracked by #13625; recorded here only as evidence. | | 6 | Under contention, updates conflict about 4.6 times for each success with 8 writers. The store has no bounded retry, even though `update` re-applies the updater to a fresh read. | Needs a decision; related to #12597. | | 7 | The concurrency harness behind these results is not in the repository, so regressions in the OCC guarantees go unnoticed. | — | Notes: - Single-threaded statement counts and throughput are on par with the pre-OCC commit. The FOR SHARE parent locks do not serialize creates. - On PostgreSQL, benchmark numbers taken right after bulk writes can be skewed by a generic plan cached before `ANALYZE` has run. Run `ANALYZE` before measuring. ### Task list - [ ] P0: Make table reads (row + columns) consistent under overwrite and drop, and keep torn reads out of the cache. Audit the other reads that span several statements, such as a role and its securable objects. - [ ] P0: Split the cascading schema/catalog soft-delete CAS into chunks inside `OccWriteSupport#deleteChildrenWithVersions`, staying in the same transaction. Cover more than 32k schemas on PostgreSQL. - [ ] P1: Stop model-version get/list from failing while versions are being linked, either with a single-statement consistent read or with a retry that runs outside the cache lock and backs off. - [ ] P1: Pass `force` as the store `cascade` flag in `MetalakeManager#dropMetalake`. - [ ] P2: Add the multi-backend OCC concurrency tests (lost update, cascade vs. create, drop vs. relation writes, rename collision, torn reads) as docker-tagged tests. - [ ] Decide whether the store should retry `update` conflicts a bounded number of times (with #12597). - [ ] Track the non-atomic default `deleteAndGet` in #13625. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
