Thank you for clarifying. Yes, You should partition your input streams as well. Typically, it does not hurt to over-partition them.
https://samza.apache.org/learn/documentation/0.13/container/samza-container.html Thanks, Jagdish On Fri, Jul 7, 2017 at 10:55 PM Angelica Garcia-Gutierrez < agar...@transcendinsights.com> wrote: > That is correct, I am augmenting each message in the input stream with > further information from the MongoDB. > > > > On 7/6/17, 5:43 PM, "Jagadish Venkatraman" <jagadish1...@gmail.com> wrote: > > >What is your input source you are reading the stream from? IIUC, you are > >trying to make a remote call to mongo to decorate/ augment each message? > > > >Best, > >Jagdish > > > >On Fri, Jul 7, 2017 at 12:51 AM Angelica Garcia-Gutierrez < > >agar...@transcendinsights.com> wrote: > > > >> Hello, > >> > >> I have a Samza job that currently makes remote calls to a MongoDB to get > >> additional information about the input stream. For scalability, MongoDB > >>was > >> initially partitioned into 4 shards (more shards will be added as > >>needed). > >> The questions are: > >> > >> * Does it make sense to attempt to partition the input stream into > >> multiple partitions such that a given task can consume it and expand the > >> message with information retrieved from a specific MongoDB shard? > >> > >> Can someone please shed some lights? > >> > >> Thanks, > >> Angelica. > >> > >> > >> The information transmitted is intended only for the person or entity to > >> which it is addressed > >> and may contain CONFIDENTIAL material. If you receive this > >> material/information in error, > >> please contact the sender and delete or destroy the > >>material/information. > > > >-- > >Sent from my iphone. > > The information transmitted is intended only for the person or entity to > which it is addressed > and may contain CONFIDENTIAL material. If you receive this > material/information in error, > please contact the sender and delete or destroy the material/information. > -- Sent from my iphone.
