shashank created CAMEL-25575:
--------------------------------
Summary: camel-dropbox - search returns only the first page of the
matches (100 by default) although its javadoc promises all files found; the
rest is dropped silently (get of a folder: first page of the listing)
Key: CAMEL-25575
URL: https://issues.apache.org/jira/browse/CAMEL-25575
Project: Camel
Issue Type: Bug
Components: camel-dropbox
Reporter: shashank
Assignee: shashank
*{{search}}* is the main case. {{DropboxAPIFacade.search}} (lines 245-259 at
main c3829069cb72), used by the {{dropbox:search}} producer and consumer:
{code:java}
SearchV2Result listing = client.files().searchV2Builder(query)
.withOptions(SearchOptions.newBuilder().withPath(remotePath).build())
.start();
return new DropboxSearchResult(listing.getMatches());
{code}
The request does not set {{max_results}}, and {{SearchOptions}} of
dropbox-core-sdk 7.1.1 uses 100 then (the constant in its no-argument
constructor, read in the jar). {{SearchV2Result}} has {{getHasMore()}} and
{{getCursor()}} for the next page ({{searchContinueV2(cursor)}}); the facade
reads neither. So with the default options a search with more matches than one
page returns the first page only: the body, and the {{FOUND_FILES}} header that
is built from it, lack the rest. Nothing is logged and the exchange succeeds.
The javadoc of the method says "@return a result object containing all the
files found." (line 242), and the component has no option or header to ask for
the next page, so a route cannot get the rest.
*{{get}} of a folder* is the lesser case ({{downloadFilesInFolder}}, lines
307-318), used by the {{dropbox:get}} producer and consumer:
{code:java}
ListFolderResult folderResult = client.files().listFolder(path.equals("/") ? ""
: path);
for (Metadata entry : folderResult.getEntries()) { ... download ... }
{code}
{{ListFolderResult}} has {{getHasMore()}} and {{getCursor()}} as well
({{listFolderContinue(cursor)}}). The facade downloads the files of the first
response only, although the javadoc says "Get the content of every file inside
the remote path" (line 296). It needs a folder larger than one page of the
listing; that size is decided by the service (the request sets no limit) and I
did not measure it. {{get}} keeps the content of every file in memory, so it is
not the operation for a very large folder anyway.
Not verified (nothing was run against Dropbox): the page size the service
really uses for a search (100 is what the SDK asks for), whether the service
limits the number of matches a search can return in total, and the page size of
a folder listing.
h3. Reproduction
New {{DropboxAPIFacadePagingTest}} (3 tests): the Dropbox client is a Mockito
mock that answers a search and a folder listing in three pages (page,
{{has_more}}, cursor; the fake fails the test after 10 continue requests). On
main 2 tests fail:
* {{searchReturnsTheMatchesOfEveryPage}}: {{the matches of the three pages,
each once and in order ==> expected: <[/docs/report-1.txt, /docs/report-2.txt,
/docs/report-3.txt, /docs/report-4.txt, /docs/report-5.txt]> but was:
<[/docs/report-1.txt, /docs/report-2.txt]>}}.
* {{getDownloadsTheFilesOfEveryPageOfTheFolder}}: {{the files of the three
pages, each once and in order ==> expected: <[/docs/a.txt, /docs/b.txt,
/docs/c.txt, /docs/d.txt]> but was: <[/docs/a.txt, /docs/b.txt]>}}.
The third, {{resultThatFitsInOnePageNeedsNoFurtherRequest}}, passes on main as
well; it is a guard that a result of one page sends no continue request (a
folder listing returns a cursor with its last page too).
h3. Proposed fix
In {{DropboxAPIFacade}}: {{search}} calls {{searchContinueV2}} with the cursor
of the response while {{getHasMore()}} is true (and a cursor is present) and
returns the matches of all pages; {{downloadFilesInFolder}} calls
{{listFolderContinue}} the same way and downloads the files of each page. The
order is the order of the service. {{mockito-core}} is added to the module as a
test dependency.
Behaviour change, with a 4.24 upgrade-guide entry: a search returns the matches
of every page, and {{get}} of a large folder downloads all its files.
Consequences that a user can notice:
* a search with many matches sends one request per page instead of one request;
the search consumer does so at every poll;
* when a request for a later page fails, the operation fails and the matches
that were already read are not returned. That is as before for a failed search,
but there are now more requests that can fail;
* the number of pages is decided by the service ({{has_more}}); the facade has
no limit of its own;
* {{get}} keeps the content of all files in the message body, as before, so a
large folder needs more memory than when only the first page was read.
No new option. camel-dropbox unit tests with the fix: 4, 0 failures (the
integration tests need a Dropbox account and were not run).
Found with a Lean 4 model of a token-paged service: the collection is a list, a
response is a {{take}} / {{drop}} slice of it plus a token when items remain.
Property: the result is the collection, every item once and in order.
Counterexample for "first page only": with a page of two, {{[1, 2, 3]}} gives
{{[1, 2]}} ({{cex_third_match_dropped}}); with a page of 100, 101 matches give
100 ({{cex_101_matches}}); {{main_complete_iff}}: the first page is the whole
result exactly when the result fits in one page. In the model, following the
token returns the whole list ({{fix_meets_spec}}) and equals main's result when
it fits in one page ({{fix_eq_main_one_page}}). The model says nothing about
the real service. Then confirmed with the real class as described.
Affected: main (4.23.0-SNAPSHOT at c3829069cb72), and the release branches
camel-4.18.x (c3fd5317a5a0), camel-4.22.x (2821f284056c) and camel-4.23.x
(197e70656a41): {{DropboxAPIFacade}} has the same lines on all three (javadoc
at 242 and 296, {{searchV2Builder}} at 252, {{listFolder}} at 309, no
{{getHasMore}} / {{getCursor}} / continue call in the file). Read, not run.
Duplicate check (2026-10-12): JIRA text "dropbox search has_more" (2,
unrelated: CAMEL-13025, CAMEL-12955), "dropbox search results truncated" (1,
unrelated: CAMEL-12955), "dropbox listFolder" (none); GitHub pull requests
"dropbox search": none about the result of a search; no open pull request
touches camel-dropbox.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
