Fatih Mehmet ARSLAN created NIFI-15657:
------------------------------------------
Summary: AzureDevOpsFlowRegistry: 409 Conflict when multiple flows
share same branch (monorepo scenario) due to branch HEAD based optimistic
locking
Key: NIFI-15657
URL: https://issues.apache.org/jira/browse/NIFI-15657
Project: Apache NiFi
Issue Type: Improvement
Components: Flow Versioning
Reporter: Fatih Mehmet ARSLAN
h3. Usage Scenario
We are using *AzureDevOpsFlowRegistry* with Apache NiFi in a monorepo-style
setup.
* Multiple independent flows exist under the same NiFi root canvas.
* All flows are versioned into the {*}same Azure DevOps repository, same
branch, same bucket{*}.
* We use a single branch (e.g. {{staging}} → {{{}prod{}}}) for version
promotion.
* We intentionally avoid per-flow branches to reduce operational overhead, as
our flows are small and managed by a small team.
This setup works well conceptually, but we are facing consistent {{409
Conflict}} errors during commit.
h3. Problem Description
When:
# Flow A commits successfully.
# Flow B (a completely different flow file in the same repo/bucket/branch)
attempts to commit.
Flow B fails with:
{{409 Conflict}}
Even if:
* Flow B performs a Pull before committing.
* The file being committed by Flow B has NOT changed.
* The last commit affecting Flow B’s file is still the same.
h3. Root Cause
Azure DevOps Push API requires {{refUpdates[].oldObjectId}} to match the
{*}current branch HEAD{*}.
However:
* NiFi tracks the {*}file-level last commit id{*}, not the branch HEAD.
* When Flow A commits, the branch HEAD changes.
* Flow B still holds the previous commit id (file-level).
* During push, Azure DevOps rejects the request because branch HEAD no longer
matches → 409.
This means independent flows inside the same branch interfere with each other,
even when modifying different files.
h3. Why Separate Branches Is Not Ideal
Using one branch per flow would technically solve this, but:
* It introduces additional repository and branch management overhead.
* For small teams managing many small flows, monorepo + single branch is
operationally simpler.
* The current behavior makes monorepo usage impractical.
h3. Requested Improvement
We would like to explore alternatives that support monorepo usage without
requiring separate branches per flow.
h3. Proposed Enhancement (Optimistic Retry at File Level)
On {{{}409 Conflict{}}}:
# Fetch current branch HEAD.
# Fetch latest commit affecting the specific file being updated.
# If the file-level commit has NOT changed:
** Retry push using the new branch HEAD as {{{}oldObjectId{}}}.
# If the file-level commit has changed:
** Fail with proper conflict message.
This would:
* Preserve safety (no overwrite if file actually changed).
* Allow concurrent commits to different flow files.
* Maintain Azure DevOps optimistic locking semantics.
* Avoid forcing branch-per-flow model.
h3. Expected Behavior
If two different flows (different files) commit sequentially to the same branch:
* Second commit should succeed
* As long as its target file has not changed
* Even if branch HEAD has moved due to other flow commits
h3. Questions for Maintainers
# Is there a recommended pattern for monorepo usage with
AzureDevOpsFlowRegistry?
# Would a file-level optimistic retry mechanism be acceptable as an
enhancement?
# Is there an alternative supported approach to avoid branch-level 409
conflicts in this scenario?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
