markhoerth opened a new issue, #14752:
URL: https://github.com/apache/gravitino/issues/14752
### Version
main branch, commit 426f8908ef69175e54f14aea04259381436c66c0 (2.0.0-SNAPSHOT)
### Describe what's wrong
Two requests that rename models in the same schema onto each other's names
at the same time (`ma` to `mb` and `mb` to `ma`) should both be refused with
409 `ModelAlreadyExistsException`, because each target name is taken. With the
PostgreSQL entity store, one of them sometimes returns 500 `RuntimeException
... ERROR: deadlock detected` instead. The same happens inside larger
concurrent workloads: the 2.0 OSS test suite saw it once in about 570,000 mixed
model creates, renames, alters, drops, loads and lists sent to two servers that
share one PostgreSQL store. Sending only the two swapping renames to one
server, about 1 in 40 rounds ends with a 500. Nothing is changed by the failed
request (PostgreSQL rolls the transaction back), but the client gets an
internal error for an ordinary conflict, and a retrying client cannot tell it
apart from a server fault.
Cause, at commit 426f8908ef:
- Model operations no longer take the tree lock
(`docs/gravitino-server-config.md:433-439`;
`core/src/main/java/org/apache/gravitino/catalog/ModelOperationDispatcher.java`
has no `TreeLockUtils` call), so the database is the only thing that orders
concurrent renames.
- A rename runs `ModelMetaService.updateModel`
(`core/src/main/java/org/apache/gravitino/storage/relational/service/ModelMetaService.java:311-367`).
Inside `doWithSchemaWriteLock` it takes the parent schema row `FOR SHARE`
(`core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/SchemaMetaPostgreSQLProvider.java:32`),
which does not serialize two writers, and then runs `UPDATE model_meta SET
model_name = ...` (`ModelMetaService.java:344-349`, SQL at
`core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/ModelMetaBaseSQLProvider.java:327-343`).
- `model_meta` has `UNIQUE (schema_id, model_name, deleted_at)`
(`scripts/postgresql/schema-2.0.0-postgresql.sql:610`). Transaction A updates
the `ma` row and then inserts the index entry for `mb`; the live `mb` row is
being updated by transaction B, so A waits for B. B updated the `mb` row and
inserts the index entry for `ma`, whose row A holds, so B waits for A.
PostgreSQL detects the cycle and aborts one of them with SQLSTATE 40P01; the
error text names `model_meta_schema_id_model_name_deleted_at_key`.
- The deadlock is not mapped to a conflict.
`ExceptionUtils.checkSQLException`
(`core/src/main/java/org/apache/gravitino/storage/relational/utils/ExceptionUtils.java:74-86`)
recognizes duplicate keys and lock-wait timeouts only; the comment at line 147
says deadlocks are left out on purpose, so that the lock order on the path gets
fixed. `PostgreSQLExceptionConverter.toGravitinoException` turns 40P01 into a
plain `IOException`
(`core/src/main/java/org/apache/gravitino/storage/relational/converters/PostgreSQLExceptionConverter.java:49`),
`ModelCatalogOperations.alterModel` wraps it as a `RuntimeException` "Failed
to load model ..."
(`catalogs/catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:325`),
and the model exception handler answers 500.
The defect is in Gravitino's own code: the rename write path takes the
unique-index entries of two rows in an order that depends on the request, and
nothing either orders them or reports the resulting deadlock as a conflict.
Possible fixes are to lock the target name before the update (for example
`SELECT ... FOR UPDATE` on the row holding the new name, or on both rows in a
fixed order such as by `model_id`), or to check for an existing model under the
new name inside the transaction and fail with `EntityAlreadyExistsException`
before the `UPDATE`.
### Error message and/or stacktrace
```
HTTP/1.1 500
{"code":1002,"type":"RuntimeException","message":"Failed to operate object
[ma] operation [ALTER] under [s1], reason [ERROR: deadlock detected\n Detail:
Process 23449 waits for ShareLock on transaction 1318008; blocked by process
23454.\nProcess 23454 waits for ShareLock on transaction 1318007; blocked by
process 23449.\n Hint: See server log for query details.\n Where: while
inserting index tuple (4735,28) in relation
\"model_meta_schema_id_model_name_deleted_at_key\"]","stack":["java.lang.RuntimeException:
Failed to load model ml1.cat_model.s1.ma: org.postgresql.util.PSQLException:
ERROR: deadlock detected", ...]}
```
The run of the test suite saw the same error on a property alter racing
other writes (`PUT .../models/m5`, `Failed to operate object [m5] operation
[ALTER] ... ERROR: deadlock detected`), with renames among the same model names
running at the same time.
### How to reproduce
Gravitino server with the relational entity store on PostgreSQL
(`gravitino.entity.store.relational.jdbcUrl = jdbc:postgresql://...`), simple
auth.
1. Create a metalake, a model catalog and a schema:
```
A="Authorization: Basic $(echo -n admin:x | base64)"
J='Content-Type: application/json'
S=http://127.0.0.1:8090/api
curl -s -H "$A" -H "$J" -X POST $S/metalakes -d
'{"name":"ml1","properties":{}}'
curl -s -H "$A" -H "$J" -X POST $S/metalakes/ml1/catalogs -d
'{"name":"cat_model","type":"MODEL","properties":{}}'
curl -s -H "$A" -H "$J" -X POST $S/metalakes/ml1/catalogs/cat_model/schemas
-d '{"name":"s1","properties":{}}'
M=$S/metalakes/ml1/catalogs/cat_model/schemas/s1/models
```
2. Register models `ma` and `mb`, send the two swapping renames at the same
time, and repeat:
```
for i in $(seq 1 100); do
for n in ma mb; do
curl -s -o /dev/null -H "$A" -X DELETE $M/$n
curl -s -o /dev/null -H "$A" -H "$J" -X POST $M -d
"{\"name\":\"$n\",\"properties\":{}}"
done
curl -s -o /tmp/r1.json -w '%{http_code}\n' -H "$A" -H "$J" -X PUT $M/ma
-d '{"updates":[{"@type":"rename","newName":"mb"}]}' &
curl -s -o /tmp/r2.json -w '%{http_code}\n' -H "$A" -H "$J" -X PUT $M/mb
-d '{"updates":[{"@type":"rename","newName":"ma"}]}' &
wait
done | sort | uniq -c
```
3. Expected: 200 responses of 409. Actual on this build: `195 409` and `5
500`; each 500 body is the deadlock error above.
### Additional context
- Found by the 2.0 OSS test suite check
`test_mixed_thrash_on_both_servers_converges` (feature: removal of the tree
lock, multi-server read/write).
- Related: #13776 (lock-wait timeouts reported as 409; it left deadlocks out
of scope and said a deadlock should be fixed by the lock order on its path),
#13731 and #13183 (tree lock removal), #13853 (EntityStore OCC hardening).
- Also present in v1.3.1: `ModelMetaBaseSQLProvider.updateModelMeta` runs
the same `UPDATE model_meta SET model_name = ...` against the same unique key,
and `ModelOperationDispatcher.alterModel` at v1.3.1 takes a WRITE tree lock
only on the source model's own name (line 277), so renames of two sibling
models do not block each other there either. That finding comes from reading
the v1.3.1 code; the race was reproduced only on the main build above.
- Only the model rename path was checked. Other entity types whose rename is
an `UPDATE` of a uniquely indexed name column without a lock on the target name
may show the same deadlock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]