Re: Architecture Bolt Design Questions

[Ambud Sharma]

Please have a look at Storm DRPC for this to see if that would help make
things easier for you.

On Mar 16, 2017 6:45 AM, "Steve Robert"  wrote:

Hi guys ,
i have an simple question with an simple scenario
 is  it   a good practice to use the bolts for this connects to an external
data source ?

Here is a concrete example:
I would connect to external API to retrieve data  for  multiple user
I have a first  Cassandra data source  which stores credential users for
each user.
I created a spout for read database and  provide a stream of this accounts

To parallelize the downloads of this user's data , I wanted to create a
Bolt with parallelism with *fieldsGrouping* strategy for avoid the process
of the same message on the same time

The Bolt could therefore consume the credential  metadata to connecte  and
get data from the  the external API.
Do you think this mechanism is acceptable ?

I know I could use a first Spout to read credential users and send it  to
Kafka
and create new Spout for read kafka , this spout could then connect to two
data sources , the first Kafka for the crendential and the external API to
have the users's datas

But I find this mechanism cumbersome for a simple scenario

Thank for your help !

-- 
Steve Robert 
Software Engineer
srob...@qualys.com
T
Qualys, Inc. – Continuous Security
Blog  | Community  |
Twitter 

Architecture Bolt Design Questions

[Steve Robert]

Hi guys ,
i have an simple question with an simple scenario
 is  it   a good practice to use the bolts for this connects to an external
data source ?

Here is a concrete example:
I would connect to external API to retrieve data  for  multiple user
I have a first  Cassandra data source  which stores credential users for
each user.
I created a spout for read database and  provide a stream of this accounts

To parallelize the downloads of this user's data , I wanted to create a
Bolt with parallelism with *fieldsGrouping* strategy for avoid the process
of the same message on the same time

The Bolt could therefore consume the credential  metadata to connecte  and
get data from the  the external API.
Do you think this mechanism is acceptable ?

I know I could use a first Spout to read credential users and send it  to
Kafka
and create new Spout for read kafka , this spout could then connect to two
data sources , the first Kafka for the crendential and the external API to
have the users's datas

But I find this mechanism cumbersome for a simple scenario

Thank for your help !

-- 
Steve Robert 
Software Engineer
srob...@qualys.com
T
Qualys, Inc. – Continuous Security
Blog  | Community  |
Twitter