abstractdog opened a new pull request, #6642:
URL: https://github.com/apache/hive/pull/6642
### What changes were proposed in this pull request?
Fix: on filesystems in UnstableRenameFileSystem (S3A/S3N/S3/GS today), the
copy suffix in Hive.mvFile carries a per-query 8-hex uniqueness tag (derived
from hive.query.id) *in place of* the numeric counter. Two concurrent writers
land at distinct destinations —
basename_copy_<queryTag1>
basename_copy_<queryTag2>
— so there is no picker loop and no rename race. On stable-rename
filesystems (HDFS, local) the historical numeric _copy_N picker is preserved
unchanged. UnstableRenameFileSystem is an in-code enum rather than a
configuration knob: the set of unsafe filesystems is a property of the
filesystem impl, not something an operator should override.
ParsedOutputFileName's copy-index regex group is widened from `[0-9]{1,6}`
to `[0-9]{1,6}|[0-9a-fA-F]{8}` so both shapes parse. getCopyIndex returns
either the numeric counter or the 8-hex tag verbatim; downstream taskId /
attemptId extraction is unaffected.
The ACID branch (taskId != -1) and the isOverwrite branch are unchanged —
ACID writers already own unique taskIds, and overwrite explicitly clears the
target first.
If a future unstable-rename filesystem is ever missed by the enum, the
failure mode is the same loud FileAlreadyExistsException → MoveTask return code
40000 that we surface today — a correct, actionable signal rather than a silent
loss.
### Why are the changes needed?
On file systems whose rename is not atomic-if-absent (S3A and other object
stores), two concurrent non-ACID INSERTs that create the same new dynamic
partition race in Hive.mvFile between the exists()-driven _copy_N picker and
the destFs.rename() call. Depending on the timing this shows up as either:
* Fail-loud — S3AFileSystem.initiateRename throws
FileAlreadyExistsException, surfacing to the client as "MoveTask return code
40000". This matches the customer report:
[load-dynamic-partitionsToAdd-0] Failed to move: ...
Caused by: FileAlreadyExistsException:
Failed to rename .../000001_N to .../000001_N_copy_M;
destination file exists
at S3AFileSystem.initiateRename
at Hive.mvFile
at Hive.copyFiles
at Hive.loadPartitionInternal
at Hive.lambda$loadDynamicPartitions
* Fail-silent — both writers' internal exists() probes see the target as
not-yet-present, both PUTs go to the same key, and the second silently
overwrites the first (last writer wins, no error surfaces).
Reproduced with 30 concurrent `insert into p_test values (i,2)` against an
S3-backed external Parquet table: 2 sessions fail with MoveTask, 6 rows
silently missing, and the final S3 listing shows the same _copy_N slot claimed
by multiple writers.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as
the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes
- provide the console output, description, screenshot and/or a reproducable
example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to
the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
* unit: TestHiveCopyFiles.testUniquenessTagAndUnstableFsGating covers the
enum recognition (matches on s3a/s3n/s3/gs, rejects hdfs/file) and the
per-query tag shape (distinct queryIds → distinct 8-hex tags, empty queryId →
empty tag). ParsedOutputFileNameTest gains 3 cases: a copy suffix that is an
8-hex tag (plain and with extension), and a strict-shape check that rejects
7-char / non-hex forms. All 31 tests green (20 in TestHiveCopyFiles under 4
parameterizations + 11 in ParsedOutputFileNameTest).
* end-to-end: 30-way concurrent burst against s3a://... table: Before: 24
rows persisted, 2 MoveTask failures, many _copy_N. After: 30 rows persisted, 0
MoveTask failures, 30 distinct 000001_N_copy_<hex> keys in S3, no FAEE.
post-patch:
```
aws s3 ls s3://dw-team-bucket/tmp/p_test/b=2/
2026-07-27 18:32:17 0
2026-07-27 18:32:45 417 000001_0_copy_0328e900
2026-07-27 18:32:31 417 000001_0_copy_0ed67c3a
2026-07-27 18:32:48 416 000001_0_copy_23f60745
2026-07-27 18:33:16 417 000001_0_copy_2d6a070a
2026-07-27 18:32:20 417 000001_0_copy_2e4c8b21
2026-07-27 18:33:22 417 000001_0_copy_2e92b5a3
2026-07-27 18:32:56 417 000001_0_copy_358e7357
2026-07-27 18:33:08 417 000001_0_copy_36aa36d2
2026-07-27 18:32:48 417 000001_0_copy_3ccf6c43
2026-07-27 18:32:42 416 000001_0_copy_4c74bc14
2026-07-27 18:32:27 417 000001_0_copy_4f457539
2026-07-27 18:32:40 417 000001_0_copy_515f0526
2026-07-27 18:32:54 417 000001_0_copy_5ae815f7
2026-07-27 18:32:34 417 000001_0_copy_61882cad
2026-07-27 18:32:19 417 000001_0_copy_73addac9
2026-07-27 18:32:39 417 000001_0_copy_79a46d49
2026-07-27 18:32:28 417 000001_0_copy_7a91235d
2026-07-27 18:33:14 417 000001_0_copy_7e9e0d2b
2026-07-27 18:33:05 417 000001_0_copy_80823534
2026-07-27 18:33:16 417 000001_0_copy_88a2d9f8
2026-07-27 18:32:27 417 000001_0_copy_8dca9748
2026-07-27 18:32:34 417 000001_0_copy_903efbf2
2026-07-27 18:32:35 417 000001_0_copy_a5906696
2026-07-27 18:32:23 417 000001_0_copy_af915863
2026-07-27 18:32:59 417 000001_0_copy_b4605e90
2026-07-27 18:33:08 417 000001_0_copy_cbb75a74
2026-07-27 18:32:50 417 000001_0_copy_e3fc1099
2026-07-27 18:32:57 417 000001_0_copy_f0c1f290
2026-07-27 18:32:19 417 000001_0_copy_f8166a50
2026-07-27 18:33:14 417 000001_0_copy_fcabab57
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
