markhoerth opened a new issue, #14752:
URL: https://github.com/apache/gravitino/issues/14752

   ### Version
   
   main branch, commit 426f8908ef69175e54f14aea04259381436c66c0 (2.0.0-SNAPSHOT)
   
   ### Describe what's wrong
   
   Two requests that rename models in the same schema onto each other's names 
at the same time (`ma` to `mb` and `mb` to `ma`) should both be refused with 
409 `ModelAlreadyExistsException`, because each target name is taken. With the 
PostgreSQL entity store, one of them sometimes returns 500 `RuntimeException 
... ERROR: deadlock detected` instead. The same happens inside larger 
concurrent workloads: the 2.0 OSS test suite saw it once in about 570,000 mixed 
model creates, renames, alters, drops, loads and lists sent to two servers that 
share one PostgreSQL store. Sending only the two swapping renames to one 
server, about 1 in 40 rounds ends with a 500. Nothing is changed by the failed 
request (PostgreSQL rolls the transaction back), but the client gets an 
internal error for an ordinary conflict, and a retrying client cannot tell it 
apart from a server fault.
   
   Cause, at commit 426f8908ef:
   
   - Model operations no longer take the tree lock 
(`docs/gravitino-server-config.md:433-439`; 
`core/src/main/java/org/apache/gravitino/catalog/ModelOperationDispatcher.java` 
has no `TreeLockUtils` call), so the database is the only thing that orders 
concurrent renames.
   - A rename runs `ModelMetaService.updateModel` 
(`core/src/main/java/org/apache/gravitino/storage/relational/service/ModelMetaService.java:311-367`).
 Inside `doWithSchemaWriteLock` it takes the parent schema row `FOR SHARE` 
(`core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/SchemaMetaPostgreSQLProvider.java:32`),
 which does not serialize two writers, and then runs `UPDATE model_meta SET 
model_name = ...` (`ModelMetaService.java:344-349`, SQL at 
`core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/ModelMetaBaseSQLProvider.java:327-343`).
   - `model_meta` has `UNIQUE (schema_id, model_name, deleted_at)` 
(`scripts/postgresql/schema-2.0.0-postgresql.sql:610`). Transaction A updates 
the `ma` row and then inserts the index entry for `mb`; the live `mb` row is 
being updated by transaction B, so A waits for B. B updated the `mb` row and 
inserts the index entry for `ma`, whose row A holds, so B waits for A. 
PostgreSQL detects the cycle and aborts one of them with SQLSTATE 40P01; the 
error text names `model_meta_schema_id_model_name_deleted_at_key`.
   - The deadlock is not mapped to a conflict. 
`ExceptionUtils.checkSQLException` 
(`core/src/main/java/org/apache/gravitino/storage/relational/utils/ExceptionUtils.java:74-86`)
 recognizes duplicate keys and lock-wait timeouts only; the comment at line 147 
says deadlocks are left out on purpose, so that the lock order on the path gets 
fixed. `PostgreSQLExceptionConverter.toGravitinoException` turns 40P01 into a 
plain `IOException` 
(`core/src/main/java/org/apache/gravitino/storage/relational/converters/PostgreSQLExceptionConverter.java:49`),
 `ModelCatalogOperations.alterModel` wraps it as a `RuntimeException` "Failed 
to load model ..." 
(`catalogs/catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:325`),
 and the model exception handler answers 500.
   
   The defect is in Gravitino's own code: the rename write path takes the 
unique-index entries of two rows in an order that depends on the request, and 
nothing either orders them or reports the resulting deadlock as a conflict. 
Possible fixes are to lock the target name before the update (for example 
`SELECT ... FOR UPDATE` on the row holding the new name, or on both rows in a 
fixed order such as by `model_id`), or to check for an existing model under the 
new name inside the transaction and fail with `EntityAlreadyExistsException` 
before the `UPDATE`.
   
   ### Error message and/or stacktrace
   
   ```
   HTTP/1.1 500
   {"code":1002,"type":"RuntimeException","message":"Failed to operate object 
[ma] operation [ALTER] under [s1], reason [ERROR: deadlock detected\n  Detail: 
Process 23449 waits for ShareLock on transaction 1318008; blocked by process 
23454.\nProcess 23454 waits for ShareLock on transaction 1318007; blocked by 
process 23449.\n  Hint: See server log for query details.\n  Where: while 
inserting index tuple (4735,28) in relation 
\"model_meta_schema_id_model_name_deleted_at_key\"]","stack":["java.lang.RuntimeException:
 Failed to load model ml1.cat_model.s1.ma: org.postgresql.util.PSQLException: 
ERROR: deadlock detected", ...]}
   ```
   
   The run of the test suite saw the same error on a property alter racing 
other writes (`PUT .../models/m5`, `Failed to operate object [m5] operation 
[ALTER] ... ERROR: deadlock detected`), with renames among the same model names 
running at the same time.
   
   ### How to reproduce
   
   Gravitino server with the relational entity store on PostgreSQL 
(`gravitino.entity.store.relational.jdbcUrl = jdbc:postgresql://...`), simple 
auth.
   
   1. Create a metalake, a model catalog and a schema:
   ```
   A="Authorization: Basic $(echo -n admin:x | base64)"
   J='Content-Type: application/json'
   S=http://127.0.0.1:8090/api
   curl -s -H "$A" -H "$J" -X POST $S/metalakes -d 
'{"name":"ml1","properties":{}}'
   curl -s -H "$A" -H "$J" -X POST $S/metalakes/ml1/catalogs -d 
'{"name":"cat_model","type":"MODEL","properties":{}}'
   curl -s -H "$A" -H "$J" -X POST $S/metalakes/ml1/catalogs/cat_model/schemas 
-d '{"name":"s1","properties":{}}'
   M=$S/metalakes/ml1/catalogs/cat_model/schemas/s1/models
   ```
   2. Register models `ma` and `mb`, send the two swapping renames at the same 
time, and repeat:
   ```
   for i in $(seq 1 100); do
     for n in ma mb; do
       curl -s -o /dev/null -H "$A" -X DELETE $M/$n
       curl -s -o /dev/null -H "$A" -H "$J" -X POST $M -d 
"{\"name\":\"$n\",\"properties\":{}}"
     done
     curl -s -o /tmp/r1.json -w '%{http_code}\n' -H "$A" -H "$J" -X PUT $M/ma 
-d '{"updates":[{"@type":"rename","newName":"mb"}]}' &
     curl -s -o /tmp/r2.json -w '%{http_code}\n' -H "$A" -H "$J" -X PUT $M/mb 
-d '{"updates":[{"@type":"rename","newName":"ma"}]}' &
     wait
   done | sort | uniq -c
   ```
   3. Expected: 200 responses of 409. Actual on this build: `195 409` and `5 
500`; each 500 body is the deadlock error above.
   
   ### Additional context
   
   - Found by the 2.0 OSS test suite check 
`test_mixed_thrash_on_both_servers_converges` (feature: removal of the tree 
lock, multi-server read/write).
   - Related: #13776 (lock-wait timeouts reported as 409; it left deadlocks out 
of scope and said a deadlock should be fixed by the lock order on its path), 
#13731 and #13183 (tree lock removal), #13853 (EntityStore OCC hardening).
   - Also present in v1.3.1: `ModelMetaBaseSQLProvider.updateModelMeta` runs 
the same `UPDATE model_meta SET model_name = ...` against the same unique key, 
and `ModelOperationDispatcher.alterModel` at v1.3.1 takes a WRITE tree lock 
only on the source model's own name (line 277), so renames of two sibling 
models do not block each other there either. That finding comes from reading 
the v1.3.1 code; the race was reproduced only on the main build above.
   - Only the model rename path was checked. Other entity types whose rename is 
an `UPDATE` of a uniquely indexed name column without a lock on the target name 
may show the same deadlock.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to