jiangzzhu opened a new pull request, #40808:
URL: https://github.com/apache/beam/pull/40808
## Summary
Makes `InitializeDoFn.createFakeParentPartition()` idempotent under
overlapping or retried `InitializeDoFn` executions during pipeline startup:
- **Problem**: Because `DoFn` execution has at-least-once semantics, two
workers can execute `InitializeDoFn` concurrently during job startup before the
first `CREATE TABLE IF NOT EXISTS` DDL operation completes. Both executions
observe `partitionMetadataDao.tableExists() == false`, both execute
`createPartitionMetadataTable()` (`CREATE TABLE IF NOT EXISTS`), and both
proceed to `createFakeParentPartition()`, causing the second execution to fail
with `SpannerException: ALREADY_EXISTS: Row [Parent0] ... already exists`.
- **Fix**:
- Inside a read-write transaction
(`partitionMetadataDao.runInTransaction`), check `transaction.getPartition(...)
== null` before inserting each initial parent partition (consistent with
`ChildPartitionsRecordAction` and `PartitionStartRecordAction`), ensuring
existing `Parent0` rows (and any state transitions already in progress) are
never overwritten.
- Also catch and ignore `SpannerException` with `ErrorCode.ALREADY_EXISTS`
as a safeguard against concurrent insert races.
## Testing
- Updated and expanded unit tests in `InitializeDoFnTest` covering new
insertions, pre-existing initial partitions (single and TVF list),
`ErrorCode.ALREADY_EXISTS` handling, and non-`ALREADY_EXISTS` exception
propagation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
