allthingssecurity opened a new pull request, #27759: URL: https://github.com/apache/camel/pull/27759
# Description [CAMEL-25575](https://issues.apache.org/jira/browse/CAMEL-25575) **`search`** (the `dropbox:search` producer and consumer) is the main case. `DropboxAPIFacade.search` returns `listing.getMatches()` of the first `search_v2` response. The request does not set `max_results`, and `SearchOptions` of dropbox-core-sdk 7.1.1 uses 100 then. `SearchV2Result.getHasMore()` and `getCursor()` are not read, so with the default options a search with more matches than one page returns the first page only; the body and the `FOUND_FILES` header lack the rest, and nothing is logged. The javadoc of the method says "@return a result object containing all the files found." (`DropboxAPIFacade.java` line 242 on main), and the component has no option or header to ask for the next page. **`get` of a folder** (the `dropbox:get` producer and consumer) is the lesser case: it downloads the files of the first `list_folder` response and does not read `ListFolderResult.getHasMore()` / `getCursor()`, although the javadoc says "Get the content of every file inside the remote path" (line 296). It needs a folder larger than one page of the listing, and `get` holds every file in memory anyway. Not verified, nothing was run against Dropbox: the page size the service really uses for a search (100 is what the SDK asks for), whether the service limits the total number of matches of a search, and the page size of a folder listing. This change: `search` calls `searchContinueV2` with the cursor of the response while `has_more` is true and returns the matches of all pages; the folder download calls `listFolderContinue` the same way. The order is the order of the service, and a result that fits in one page needs no further request. `mockito-core` is added to the module as a test dependency. What a user can notice (the 4.24 upgrade guide entry has all of it but the fourth point): - a search returns the matches of every page, and `get` of a large folder downloads all its files; - a search with many matches sends one request per page instead of one; the search consumer does so at every poll; - when a request for a later page fails, the operation fails and the matches already read are not returned. A failed search lost its result before as well, but there are now more requests that can fail; - the number of pages is decided by the service (`has_more`); the facade has no limit of its own; - `get` keeps the content of all files in the message body, as before, so a large folder needs more memory than when only the first page was read. The 4_24 guide is not on main (c3829069cb72), so this PR creates it with the header of the 4_23 guide and adds it to the list in `camel-4x-upgrade-guide.adoc`. Tests: - New `DropboxAPIFacadePagingTest` (3 tests). The Dropbox client is a Mockito mock that answers a search and a folder listing in three pages (page, `has_more`, cursor); it fails the test after 10 continue requests. - Without the change 2 of them fail: `the matches of the three pages, each once and in order ==> expected: <[/docs/report-1.txt, /docs/report-2.txt, /docs/report-3.txt, /docs/report-4.txt, /docs/report-5.txt]> but was: <[/docs/report-1.txt, /docs/report-2.txt]>` and `the files of the three pages, each once and in order ==> expected: <[/docs/a.txt, /docs/b.txt, /docs/c.txt, /docs/d.txt]> but was: <[/docs/a.txt, /docs/b.txt]>`. - The third passes without the change as well. It is a guard that a result of one page sends no continue request (a folder listing returns a cursor with its last page too). - With the change, the camel-dropbox unit tests pass: 4 tests, 0 failures. The integration tests need a Dropbox account and were not run. Found with a Lean 4 model of a token-paged service: the collection is a list, a response is a `take` / `drop` slice of it plus a token when items remain. Property: the result is the collection, every item once and in order. Counterexample for "first page only": with a page of two, `[1, 2, 3]` gives `[1, 2]`. In the model, following the token returns the whole list, and the same as main when the result fits in one page. The model says nothing about the real service. I then reproduced it with the real class. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested the affected module, including the formatter and import-sort plugins. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
