markhoerth opened a new issue, #14420:
URL: https://github.com/apache/gravitino/issues/14420
### Version
main branch, commit bb634defdc5663a8124be5fca1c8d9e52ac096ac (2.0.0-SNAPSHOT)
### Describe what's wrong
Clients create and drop the same table name concurrently. Each create should
either succeed (200) or report that the name is taken (409). Some creates
return 404 "Metadata object ... or owner admin is not found" or 404 "The
metadata object of ... isn't found". The table was in fact created; a
concurrent drop removed it before the create request finished. The client is
told the create failed with "not found", which does not describe what happened
and does not say whether a retry is safe.
The cause is in
`core/src/main/java/org/apache/gravitino/hook/TableHookDispatcher.java:87-99`.
`createTable` first creates the table (`:87-89`), then sets the creator as
owner (`:94-98`) as a separate step. If a drop of the same name lands between
the two steps, `OwnerManager.setOwner`
(`core/src/main/java/org/apache/gravitino/authorization/OwnerManager.java:72`)
finds no entity: `getOwner` throws `NoSuchMetadataObjectException` at `:318`,
or the store throws `NoSuchEntityException`, which `setOwner` rethrows as
`NotFoundException` at `:113-117`. The exception propagates out of
`createTable` and the REST layer reports a 404 for the create.
### Error message and/or stacktrace
```
POST /api/metalakes/ml/catalogs/cat/schemas/s/tables
404 {"code":1003,"type":"NotFoundException","message":"Failed to operate
table(s) [x] operation [CREATE] under schema [s], reason [Metadata object
cat.s.x or owner admin is not found]"}
404 {"code":1003,"type":"NoSuchMetadataObjectException","message":"Failed to
operate table(s) [x] operation [CREATE] under schema [s], reason [The metadata
object of cat.s.x isn't found]"}
org.apache.gravitino.exceptions.NoSuchMetadataObjectException: The metadata
object of cat.s.x isn't found
at
org.apache.gravitino.authorization.OwnerManager.getOwner(OwnerManager.java:318)
at
org.apache.gravitino.authorization.OwnerManager.setOwner(OwnerManager.java:72)
at
org.apache.gravitino.hook.TableHookDispatcher.createTable(TableHookDispatcher.java:94)
at
org.apache.gravitino.catalog.TableNormalizeDispatcher.createTable(TableNormalizeDispatcher.java:82)
at
org.apache.gravitino.listener.TableEventDispatcher.createTable(TableEventDispatcher.java:152)
at
org.apache.gravitino.server.web.rest.TableOperations.lambda$createTable$1(TableOperations.java:153)
```
### How to reproduce
1. Create a metalake, an Iceberg catalog and a schema:
```
H='-u admin:x -H Content-Type:application/json -H
Accept:application/vnd.gravitino.v1+json'
curl $H -X POST http://127.0.0.1:8090/api/metalakes -d
'{"name":"ml","comment":"","properties":{}}'
curl $H -X POST http://127.0.0.1:8090/api/metalakes/ml/catalogs -d
'{"name":"cat","type":"RELATIONAL","provider":"lakehouse-iceberg","comment":"","properties":{"catalog-backend":"memory","uri":"memory","warehouse":"/tmp/ml_cat"}}'
curl $H -X POST
http://127.0.0.1:8090/api/metalakes/ml/catalogs/cat/schemas -d
'{"name":"s","comment":"","properties":{}}'
```
2. Save this script as `race.sh`:
```
#!/bin/bash
B=http://127.0.0.1:8090/api/metalakes/ml/catalogs/cat/schemas/s/tables
if [ "$1" = c ]; then
curl -s -o /dev/null -w 'create %{http_code}\n' -u admin:x -H
'Content-Type: application/json' -H 'Accept: application/vnd.gravitino.v1+json'
-X POST $B -d
'{"name":"x","comment":"","columns":[{"name":"id","type":"long","nullable":true}],"properties":{}}'
else
curl -s -o /dev/null -w 'drop %{http_code}\n' -u admin:x -H 'Accept:
application/vnd.gravitino.v1+json' -X DELETE $B/x
fi
```
3. Run 150 creates and 150 drops, 16 at a time:
```
chmod +x race.sh; for i in $(seq 1 150); do echo c; echo d; done | xargs
-P 16 -n1 ./race.sh | sort | uniq -c
```
In one run, 34 of the 150 creates returned 404.
### Additional context
Related: #13730 and #13733 (concurrency gaps around TreeLock removal);
neither names this case. The 2.0 OSS test suite check
`test_drop_and_create_racing_on_one_name_stay_consistent` found it. The defect
is in Gravitino's create-then-set-owner sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
