Rarasinhoo commented on PR #11913:
URL: https://github.com/apache/nifi/pull/11913#issuecomment-6926644855

   Measurement on a production-shaped flow, for reviewers who want numbers.
   
   **Setup.** Synthetic flows reproducing the nesting of a real installation 
(1,192 versioned Process Groups, 76 top-level, nesting up to 7 levels), stored 
in a private GitHub repository. A sample of 4 top-level trees (73 versioned 
Process Groups, nesting up to 6 levels) was imported through the REST API into 
two local builds: `main` at 659bb60c, and the same commit with #11913, #11925 
and #11926 applied. GitHub requests were counted with a proxy between NiFi and 
the API. Personal Access Token (5,000 requests/hour), Synchronization Interval 
1 min.
   
   | | `main` | with #11913, #11925, #11926 |
   |---|---|---|
   | Import of the 4 trees | not finished after 22 min and 4,920 requests; the 
token limit was exhausted (403) | 444 requests, 103 s |
   | Synchronization after import | not reached | 79 requests in 5 min: 74 
snapshot reads right after the import, then 1 branch-head check per minute |
   
   On `main`, the 4,920 requests were about 1,228 flow fetches, each costing 
contents + blob + branch ref + commit listing. With the changes each nested 
flow is fetched once, and while the branch head is unchanged a synchronization 
costs one request for the whole branch, independent of the number of Process 
Groups. The synthetic repository has only a few commits, so the commit-author 
lookups that `main` does per uncached commit were mostly cache hits here; on a 
repository with long history `main` costs more per fetch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to