[
https://issues.apache.org/jira/browse/CAMEL-15718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17233444#comment-17233444
]
Erik Zimmermann edited comment on CAMEL-15718 at 11/17/20, 8:41 AM:
--------------------------------------------------------------------
Thanks a lot [~zbendhiba] . I'm not sure if this fix is able to handle
processing mass data, e.g., coming from external filebeat instances:
# If all threads in a session have finished processing the current batches of
data and the last one finally passes phaser.arriveAndDeregister(), the phaser
itself terminates permanently. As I understand the phaser, it does not accept
any further registration and stops its service from that point on. Thus, data
arriving afterwards will be treated as without the fix.
# If a thread passes the phaser.arriveAndDeregister(), all waiting threads at
the phaser.arriveAndAwaitAdvance() start running all at once and not one after
another. As a result the window.size is concurrently messed up again.
# I don't clearly understand the flush pause during the Ack which might also
cause performance reduction?
Is it possible to teach camel to also use the LumberjackSessionHandler multi
threaded correctly? This would prevent all the messy behaviour of the handler.
was (Author: ezett):
Thanks a lot [~zbendhiba] . I'm not sure if this fix is able to handle
processing mass data, e.g., coming from external filebeat instances:
# If all threads in a session have finished processing the current batches of
data and the last one finally passes phaser.arriveAndDeregister(), the phaser
itself terminates permanently. As I understand the phaser, it does not accept
any further registration and stops its service from that point on. Thus, data
arriving afterwards will be treated as without the fix.
# If a thread passes the phaser.arriveAndDeregister(), all waiting threads at
the phaser.arriveAndAwaitAdvance() start running all at once and not one after
another. As a result the window.size is concurrently messed up again.
# I don't clearly understand the flush pause during the Ack which might also
cause performance reduction?
Is it possible to teach camel to also use the LumberjackSessionHandler multi
threaded correctly? This would prevent all the messy behaviour of the handler.
> Camel lumberjack server component not thread safe
> -------------------------------------------------
>
> Key: CAMEL-15718
> URL: https://issues.apache.org/jira/browse/CAMEL-15718
> Project: Camel
> Issue Type: Bug
> Affects Versions: 3.5.0, 3.6.0
> Environment: camel 3.5.0, filebeat 7.9.1, jdk 1.8.0_242, Debian 10,
> Linux 4.19.0-6-amd64
> Reporter: Erik Zimmermann
> Assignee: Zineb BENDHIBA
> Priority: Major
> Fix For: 3.7.0
>
> Attachments: filebeat.CAMEL-15718.yml
>
>
> I am new to camel. I use a filebeat source that delivers logfile data over
> the lumberjack v2 batch protocol. As a receiving server I use the camel
> lumberjack component to further process the data in a camel pipeline.
> I realized that the LumberjackSessionHandler of camels lumberjack component
> is not stateless but is being used by camel for all parallel lumberjack
> connection requests. Thus, new incoming connections with their own batch
> windows mess up any ongoing process of the already existing window, e.g.
> window size settings and counting acknowledges.
> Many different combinations of multicast().parallelProcessing().threads(...)
> with filebeats workers/pipelines showed the same result.
> The states of the stateful LumberjackSessionHandler are mixed up and the
> handling of parallel windows is broken which results in unwanted/unfinished
> acknowledges. As a result it hangs up the whole communication process to the
> filebeat source.
> The LumberjackSessionHandler is not able to handle multiple threads, but
> camel uses it as it would be able to.
> How can I tell camel to use separate LumberjackSessionHandlers and processing
> pipelines for each lumberjack batch request? Or do I misunderstand the
> concept of how camel uses components?
> Sorry, I wasn't able to find a more specific "camel-lumberjack" component in
> the list of issue proposals above so I selected camel-core...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
