[
https://issues.apache.org/jira/browse/NIFI-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard reassigned NIFI-15657:
-------------------------------------
Assignee: Pierre Villard
> AzureDevOpsFlowRegistry: 409 Conflict when multiple flows share same branch
> (monorepo scenario) due to branch HEAD based optimistic locking
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-15657
> URL: https://issues.apache.org/jira/browse/NIFI-15657
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Flow Versioning
> Reporter: Fatih Mehmet ARSLAN
> Assignee: Pierre Villard
> Priority: Major
>
> h3. Usage Scenario
> We are using *AzureDevOpsFlowRegistry* with Apache NiFi in a monorepo-style
> setup.
> * Multiple independent flows exist under the same NiFi root canvas.
> * All flows are versioned into the {*}same Azure DevOps repository, same
> branch, same bucket{*}.
> * We use a single branch (e.g. {{staging}} → {{{}prod{}}}) for version
> promotion.
> * We intentionally avoid per-flow branches to reduce operational overhead,
> as our flows are small and managed by a small team.
> This setup works well conceptually, but we are facing consistent {{409
> Conflict}} errors during commit.
> h3. Problem Description
> When:
> # Flow A commits successfully.
> # Flow B (a completely different flow file in the same repo/bucket/branch)
> attempts to commit.
> Flow B fails with:
>
> {{409 Conflict}}
> Even if:
> * Flow B performs a Pull before committing.
> * The file being committed by Flow B has NOT changed.
> * The last commit affecting Flow B’s file is still the same.
> h3. Root Cause
> Azure DevOps Push API requires {{refUpdates[].oldObjectId}} to match the
> {*}current branch HEAD{*}.
> However:
> * NiFi tracks the {*}file-level last commit id{*}, not the branch HEAD.
> * When Flow A commits, the branch HEAD changes.
> * Flow B still holds the previous commit id (file-level).
> * During push, Azure DevOps rejects the request because branch HEAD no
> longer matches → 409.
> This means independent flows inside the same branch interfere with each
> other, even when modifying different files.
> h3. Why Separate Branches Is Not Ideal
> Using one branch per flow would technically solve this, but:
> * It introduces additional repository and branch management overhead.
> * For small teams managing many small flows, monorepo + single branch is
> operationally simpler.
> * The current behavior makes monorepo usage impractical.
> h3. Requested Improvement
> We would like to explore alternatives that support monorepo usage without
> requiring separate branches per flow.
> h3. Proposed Enhancement (Optimistic Retry at File Level)
> On {{{}409 Conflict{}}}:
> # Fetch current branch HEAD.
> # Fetch latest commit affecting the specific file being updated.
> # If the file-level commit has NOT changed:
> ** Retry push using the new branch HEAD as {{{}oldObjectId{}}}.
> # If the file-level commit has changed:
> ** Fail with proper conflict message.
> This would:
> * Preserve safety (no overwrite if file actually changed).
> * Allow concurrent commits to different flow files.
> * Maintain Azure DevOps optimistic locking semantics.
> * Avoid forcing branch-per-flow model.
> h3. Expected Behavior
> If two different flows (different files) commit sequentially to the same
> branch:
> * Second commit should succeed
> * As long as its target file has not changed
> * Even if branch HEAD has moved due to other flow commits
> h3. Questions for Maintainers
> # Is there a recommended pattern for monorepo usage with
> AzureDevOpsFlowRegistry?
> # Would a file-level optimistic retry mechanism be acceptable as an
> enhancement?
> # Is there an alternative supported approach to avoid branch-level 409
> conflicts in this scenario?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
