Hisao Furuichi created CAMEL-12179:
--------------------------------------
Summary: Implement Camel Aws-Kinesis Using Kinesis Client
Library(KCL)
Key: CAMEL-12179
URL: https://issues.apache.org/jira/browse/CAMEL-12179
Project: Camel
Issue Type: New Feature
Components: camel-aws
Affects Versions: 2.17.0
Environment: * JBoss Fuse 6.3 R5
** camel 2.17.0
Reporter: Hisao Furuichi
Amazon provides 2 options for Kinesis Java Client:
* *Kinesis Client Library(KCL)*
* *Kinesis Streams API*
"*Kinesis Streams API*" is a low level API set, and "*KCL*" is a rich library.
In the official document, "*KCL*" is described like this:
{quote}
The KCL is compiled into your application to enable fault-tolerant consumption
of data from the stream.
The KCL ensures that for every shard there is a record processor running and
processing that shard.
The library also simplifies reading data from the stream.
The KCL uses an Amazon DynamoDB table to store control data. It creates one
table per application that is processing data.
The KCL takes care of many of the complex tasks associated with distributed
computing, such as load-balancing across multiple instances, responding to
instance failures, checkpointing processed records, and reacting to resharding.
The KCL enables you to focus on writing record processing logic.
{quote}
Since "*Kinesis Streams API*" is used to implement current aws-kinesis
component, usability and scalability are not provided by default.
* User needs to specify *shardID*(ENTESB-7517) which might be changed
dynamically
* If a Stream uses multiple shards, user needs to create multiple endpoints for
it. (Increasing number of shards are tips to scale out. If there are 200
shards, user needs to maintain 200 endpoints)
* If user wants to consume a same shard from multiple Camel runtimes, user
needs to implement load-balancing
By using "*KCL*" to implement *aws-kinesis*, above current difficulties/concern
can be resolved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
