aiskakov28 opened a new issue, #18759: URL: https://github.com/apache/iceberg/issues/18759
### Apache Iceberg version 1.10.1 (`apache/iceberg-rest-fixture:1.10.1`). Also seen with iceberg-core 1.11.0, and the code is the same on `main` (`3ad26de`). ### Query engine None (REST API directly) ### Please describe the bug 🐞 If a few clients create the same new namespace at the same time, one of them gets a 200 and most of the others get the expected 409, but some get a 500: ``` UncheckedSQLException: Failed to execute: INSERT INTO iceberg_namespace_properties (catalog_name, namespace, property_key, property_value) VALUES (?,?,?,?) Caused by: org.sqlite.SQLiteException: [SQLITE_CONSTRAINT_PRIMARYKEY] A PRIMARY KEY constraint failed (UNIQUE constraint failed: iceberg_namespace_properties.catalog_name, iceberg_namespace_properties.namespace, iceberg_namespace_properties.property_key) ``` The REST spec gives `409: Conflict - The namespace already exists` for this case, and that's what a create sent after the first one has finished gets. Nothing is damaged: there's always exactly one winner, and the namespace that ends up stored has only the winner's properties. The problem is just the status. A 500 tells the client it doesn't know whether its create worked, while the catalog actually knows: the namespace exists and this request created nothing. It isn't specific to SQLite. Gravitino's Iceberg REST server, which uses `JdbcCatalog` with Postgres, does the same thing: ``` Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "iceberg_namespace_properties_pkey" ``` With 6 concurrent creates per round and 20 rounds (120 creates): | Catalog | 200 | 409 | 500 | | --- | ---: | ---: | ---: | | `apache/iceberg-rest-fixture:1.10.1` (SQLite), 4 runs | 20 | 72–93 | 7–28 | | `apache/iceberg-rest-fixture:latest`, built from `main` at `b0df3ca` (SQLite) | 20 | 84 | 16 | | Gravitino 1.3.1 with iceberg-core 1.11.0 (Postgres 17.6), 2 runs | 20 | 69–77 | 23–31 | **Cause** `createNamespace` checks `namespaceExists` and then inserts the properties as a separate step: https://github.com/apache/iceberg/blob/3ad26de2bd62600724d38d3428f65f4f7cde2586/core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java#L461-L478 Concurrent creates can all pass the check. The first insert wins, and the others hit the primary key on `iceberg_namespace_properties`: https://github.com/apache/iceberg/blob/3ad26de2bd62600724d38d3428f65f4f7cde2586/core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java#L887-L907 The insert goes through `execute` with an error handler that does nothing, so the duplicate key, like any other `SQLException`, comes out as `UncheckedSQLException`, which REST servers return as a 500: https://github.com/apache/iceberg/blob/3ad26de2bd62600724d38d3428f65f4f7cde2586/core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java#L809-L831 Creates that run their check after the winner's insert get the normal 409, which is why each round has a mix of both. The insert is a single statement, so a losing create doesn't write any of its properties. **Others have run into this** The Go implementation's SQL catalog had the same check-then-insert race and fixed it in apache/iceberg-go#1635, by checking again whether the namespace exists after a failed insert. bruin-data/ingestr#1056 hit this Java version through a REST catalog, with the same `UncheckedSQLException` on `INSERT INTO iceberg_namespace_properties`. It made a first run fail when two jobs wrote to a new namespace at once, and they worked around it on their side. I couldn't find an existing report here. One possible fix (I haven't tried it here): if the insert fails, check again whether the namespace exists, and throw `AlreadyExistsException` if it does. Otherwise rethrow the original error as before. That works the same on every database, without matching database-specific error codes. It's the approach iceberg-go took for the same race in its SQL catalog (apache/iceberg-go#1635). **To reproduce** ```sh docker run -d -p 8181:8181 apache/iceberg-rest-fixture:1.10.1 ./repro.sh http://localhost:8181 ``` ``` ... round 19: 409 409 500 200 409 409 round 20: 500 200 500 409 409 409 200: 20 409: 72 500: 28 Caused by: org.sqlite.SQLiteException: [SQLITE_CONSTRAINT_PRIMARYKEY] A PRIMARY KEY constraint failed (...) ``` <details><summary><code>repro.sh</code> (bash, curl, jq)</summary> ```sh #!/usr/bin/env bash # Concurrent creates of one new namespace. One should get 200 and the rest 409; # prints how many got each status, and the cause of any 500. # usage: ns-repro.sh CATALOG_URL [ROUNDS] [CLIENTS] (e.g. http://localhost:8181) # Exit status: 3 if any create got a 5xx, 0 if none, 1 on anything unexpected. set -euo pipefail url=${1%/}; rounds=${2:-20}; clients=${3:-6} dir=$(mktemp -d); id=$(date +%s) fail() { echo "error: $*" >&2; exit 1; } post() { # out-file namespace curl -sS -m 30 -o "$1" -w '%{http_code}' -H 'Content-Type: application/json' \ -X POST "$url/v1/namespaces" -d "{\"namespace\":[\"$2\"]}" 2>/dev/null || true } for r in $(seq "$rounds"); do ns="ns_race_${id}_$r" for c in $(seq "$clients"); do post "$dir/$r.$c" "$ns" > "$dir/$r.$c.code" & done wait codes=$(for c in $(seq "$clients"); do cat "$dir/$r.$c.code"; echo; done) [ "$(grep -c '^200$' <<<"$codes")" = 1 ] || fail "round $r: expected one 200, got: $(echo $codes)" grep -qvE '^(200|409|5[0-9][0-9])$' <<<"$codes" && fail "round $r: unexpected status: $(echo $codes)" [ "$(curl -sS -m 30 -o /dev/null -w '%{http_code}' "$url/v1/namespaces/$ns")" = 200 ] \ || fail "round $r: namespace $ns not found afterwards" echo "round $r: $(echo $codes)" done for f in "$dir"/*.code; do cat "$f"; echo; done | sort | uniq -c | awk '{print " " $2 ": " $1}' for f in "$dir"/*.code; do if [[ $(cat "$f") == 5* ]]; then jq -r '.error.stack[]? | select(startswith("Caused by"))' "${f%.code}" | head -1; break fi done grep -q '^5' "$dir"/*.code && exit 3 || exit 0 ``` </details> ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time Found by @aiskakov28 and @aldusialn. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
