shashank created CAMEL-25548:
--------------------------------
Summary: camel-twitter - twitter-search ignores filterOld,
sinceId, lang, count and the geolocation options since the Twitter4J 4.1
upgrade, so the consumer routes the same tweets again at every poll; a
twitter-timeline poll sends its request twice
Key: CAMEL-25548
URL: https://issues.apache.org/jira/browse/CAMEL-25548
Project: Camel
Issue Type: Bug
Components: camel-twitter
Reporter: shashank
Assignee: shashank
Since Twitter4J 4.1 a {{twitter4j.v1.Query}} is immutable: {{sinceId}},
{{maxId}}, {{lang}}, {{count}} and {{geoCode}} return a new {{Query}}. When
Camel 4.0.0 moved to Twitter4J 4.1.2, the setters in {{SearchConsumerHandler}}
and {{SearchProducer}} were renamed, but the results are not used (lines at
main 9432ba16d026: {{SearchConsumerHandler}} 66, 88, 92, 103;
{{SearchProducer}} 66, 75, 81, 91, 100):
{code:java}
Query query = Query.of(keywords);
if (endpoint.getProperties().isFilterOld()) {
query.sinceId(getLastId()); // returns a new Query, which is dropped
}
...
query.lang(endpoint.getProperties().getLang());
query.count(endpoint.getProperties().getCount());
query.geoCode(location, radius, unit);
QueryResult qr = twitter.v1().search().search(query); // only the keywords
{code}
So the query that is sent has only the keywords:
* The consumer ignores {{filterOld}} (default {{true}}): every poll (default
every 30 seconds) gets the most recent tweets of the search again and routes
them again. Nothing is logged.
* The consumer ignores {{lang}}, {{count}} (default 5; Twitter's default of 15
applies) and {{latitude}}/{{longitude}}/{{radius}}/{{distanceMetric}}.
* The producer ignores {{sinceId}}, {{lang}}, {{count}} and the headers
{{CamelTwitterSinceId}}, {{CamelTwitterMaxId}}, {{CamelTwitterSearchLanguage}}
and {{CamelTwitterCount}}.
A second defect in the same class, from a later cleanup (c261528d20, 4.1.0):
{{SearchConsumerHandler.directConsume}} returns an empty list when keywords are
given, because the blank check was inverted ({{keywords == null ||
!keywords.isBlank()}}), so a consumer with {{type=direct}} never routes
anything at startup.
A third defect in the same module, in the timeline consumers (since
CAMEL-15651, which introduced the class):
{{AbstractStatusConsumerHandler.pollConsume}} (lines 38-45), used by the home,
mentions, user and list timelines, calls {{doPoll()}}, the request to Twitter,
twice:
{code:java}
List<Status> statusList = doPoll();
for (Status status : doPoll()) {
setLastIdIfGreater(status.getId());
}
return TwitterEventType.STATUS.createExchangeList(endpoint, statusList);
{code}
The tweets of the first response are routed, and the second response is only
used for the id of the last tweet, which the next poll sends as {{since_id}}.
Every poll sends two requests where one is enough (the home timeline allows 15
requests per 15 minutes), and a tweet posted between the two requests is only
in the second response, so it is skipped.
h3. Reproduction
New {{TwitterSearchQueryOptionsTest}} (no network: a mocked {{Twitter}} client
is set on the endpoint with {{TwitterConfiguration.setTwitter}}; its search
records the query and returns the tweets with an id above {{since_id}} and up
to {{max_id}}, as the search API does; the consumer is created with
{{startScheduler=false}} and polled directly). On main, twice, five of the six
tests fail:
{noformat}
consumerPollsOnlyTheTweetsNewerThanTheLastOne: The second poll must not route
the same tweets again ==> expected: <[10, 11]> but was: <[10, 11, 10, 11]>
consumerSendsLanguageCountAndGeolocation: expected: <en> but was: <null>
directConsumerSearchesTheKeywordsAtStartup: expected: <[10, 11]> but was:
<[]>
producerSendsSinceIdMaxIdLanguageAndCountOfTheHeaders: expected: <[11, 12]> but
was: <[10, 11, 12, 13]>
producerSendsTheOptionsOfTheEndpoint: expected: <1> but was: <3>
{noformat}
The sixth test ({{filterOld=false}} polls the tweets again) passes on main too.
New {{TwitterTimelinePollTest}} (same setup; the home timeline of the mocked
client returns the tweets above {{since_id}}). On main both tests fail:
{noformat}
pollSendsOneRequest: A poll must send one request to the timeline ==> expected:
<1> but was: <2>
tweetPostedDuringAPollIsRoutedByTheNextPoll: The tweet posted during the first
poll must not be skipped ==> expected: <[10, 11]> but was: <[10]>
{noformat}
h3. Proposed fix
Keep the {{Query}} that each call returns ({{query = query.sinceId(...)}} and
so on), in the consumer handler and in the producer, and correct the blank
check of {{directConsume}}. The options then work as documented and as in Camel
3.x. Visible to users: a poll routes only tweets newer than the last one
consumed, and a page has {{count}} tweets (default 5), so the 4.23 upgrade
guide has a note (raise {{count}} and {{numberOfPages}} for a search that
matches more tweets between two polls). In
{{AbstractStatusConsumerHandler.pollConsume}} the loop uses the list of the
first, and now only, request (no upgrade note: same tweets, one request per
poll). camel-twitter with the fix: 14 unit tests, 0 failures; the integration
tests need Twitter credentials and were not run. Test dependency added:
mockito-junit-jupiter.
Not changed here: in the producer, {{filterOld}} is overridden by {{sinceId}}
(default 1) as it was in Camel 3.x; and
{{AbstractTwitterConsumerHandler.getLastIdPaging}} passes {{numberOfPages}} to
{{Paging.ofPage}}, so {{numberOfPages=3}} requests the third page only (the
default of 1 is not affected).
Found with a search for calls whose result is dropped on immutable types, and a
small Lean 4 model of the poll (the matching tweets as a list of ids, the API
as "ids above since_id and up to max_id"): on main a second poll with no new
tweet is proved to route all the tweets again, for every list of tweets; with
the fix it is proved to route nothing, and a poll routes exactly the tweets
above the last id. Confirmed with the real consumer and producer.
Affected: main, camel-4.22.x, camel-4.18.x, camel-4.14.x (same code, GitHub
contents API); the query options since 4.0.0, the {{type=direct}} check since
4.1.0, the double timeline request in all of them.
Duplicate check (2026-10-11): JIRA text "twitter" since 2023-06 (6, none about
search), "twitter-search" (30, none since the 4.0 upgrade), "twitter4j" (22,
none after 2022); GitHub pull requests with "twitter" in the title: none open.
No open pull request changes {{SearchConsumerHandler}}, {{SearchProducer}} or
{{AbstractStatusConsumerHandler}}.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
